Robots.txt

I was looking my google search console, I found there are more than 1.5 excluded urls :|
all of them affected by
Disallow: /posts/

Last crawl
Sep 25, 2018, 8:47:32 AM
Crawl allowed?No: blocked by robots.txt
Page fetch
Failed: Blocked by robots.txt

Whats your opinion?

Should I delete that line from robot.txt ?
 
Twitter is not showing meta images on my links.
The error says
what should I do

1639250989503.png

INFO: Page fetched successfully
INFO: 18 metatags were found
INFO: twitter:card = summary tag found
INFO: Card loaded successfully
WARN: The image URL https://www.maasmutemeti.com/forum/data/assets/logo/face_logo_uzman.png specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

robot.txt

User-agent: *
Allow: /

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/ajax/
Disallow: /forum/misc/contact/
Disallow: /forum/data/
Disallow: /forum/conversations/
Disallow: /forum/events/birthdays/
Disallow: /forum/events/monthly/
Disallow: /forum/events/weekly/
Disallow: /forum/find-new/
Disallow: /forum/help/
Disallow: /forum/internal_data/
Disallow: /forum/js/
Disallow: /forum/library/
Disallow: /forum/search/
Disallow: /forum/styles/
Disallow: /forum/login/
Disallow: /forum/lost-password/
Disallow: /forum/online/
Allow: /
Sitemap: https://www.maasmutemeti.com/forum/sitemap.php
 
Would one disallow their off topic forum? Pros? Cons?
No, I just noindex the forums (in the forum settings) I don’t want crawled and indexed. Robots.txt disallow doesn’t necessarily stop indexing.
 
Last edited:
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot 
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: YandexBot
User-agent: DotBot
User-agent: omgilibot
User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Barkrowler
Disallow: /


User-agent: Mediapartners-Google
Allow: /

User-agent: *
Disallow: /conversations/
Disallow: /forums/*?prefix_id
Disallow: /forums/*?order
Disallow: /forums/*?unanswered
Disallow: /forums/*?unsolved
Disallow: /threads/*?order
Disallow: /media/*?order
Disallow: /forums/*?filter_threads=
Disallow: /media/*?no_date_limit=
Disallow: /misc/language
Disallow: /misc/style
Disallow: /mailto:
Disallow: /tel:

Removed pages with noindex on them. Added all filters.

Removed nofollow on links to let google crawl because sometimes it indexes them. Nofollow (or ugc) is also propably best used only on external links.

Those 3 bots had to be added separately to work.
 
Last edited:
Back
Top Bottom