Robots.txt

[Asser]

Does the robot.txt file go in your public_html folder

For sure!
Upload it on public_html

 

[Gregsta]

For sure!
Upload it on public_html

Thanks for that

 

[dethfire]

Noticed Bing was indexing /search/ results so I disallowed them now.

 

[boeing747]

I was looking my google search console, I found there are more than 1.5 excluded urls :|
all of them affected by
Disallow: /posts/

Last crawl
Sep 25, 2018, 8:47:32 AM
Crawl allowed?No: blocked by robots.txt
Page fetch
Failed: Blocked by robots.txt

Whats your opinion?

Should I delete that line from robot.txt ?

 

[boeing747]

any suggestion?

 

[MySiteGuy]

any suggestion?

See post #68 of this thread.

 

[ERSAN.NET]

hello, which robots.txt file would you suggest?

 

[KSA]

Is there away to disallow Google bots crawling/seeing/recgonizing external links?

 

[htsumer]

Twitter is not showing meta images on my links.
The error says
what should I do

INFO: Page fetched successfully
INFO: 18 metatags were found
INFO: twitter:card = summary tag found
INFO: Card loaded successfully
WARN: The image URL https://www.maasmutemeti.com/forum/data/assets/logo/face_logo_uzman.png specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

robot.txt

User-agent: *
Allow: /

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/ajax/
Disallow: /forum/misc/contact/
Disallow: /forum/data/
Disallow: /forum/conversations/
Disallow: /forum/events/birthdays/
Disallow: /forum/events/monthly/
Disallow: /forum/events/weekly/
Disallow: /forum/find-new/
Disallow: /forum/help/
Disallow: /forum/internal_data/
Disallow: /forum/js/
Disallow: /forum/library/
Disallow: /forum/search/
Disallow: /forum/styles/
Disallow: /forum/login/
Disallow: /forum/lost-password/
Disallow: /forum/online/
Allow: /
Sitemap: https://www.maasmutemeti.com/forum/sitemap.php

 

[htsumer]

Twitter is not showing meta images on my links.
The error says
what should I do

View attachment 261783



robot.txt

current

 

[Jeremy P]

Twitter is not showing meta images on my links.
The error says
what should I do

View attachment 261783



robot.txt

You have Disallow: /forum/data/ in your robots.txt. Either remove that line or add additional rules to let certain bots bypass it.

 

[htsumer]

You have Disallow: /forum/data/ in your robots.txt. Either remove that line or add additional rules to let certain bots bypass it.

Thank you. Deleted it fixed it

 

[click here]

Would one disallow their off topic forum? Pros? Cons?

 

[snoopy5]

For what is this for?

Disallow: /forums/-/

 

[Mr Lucky]

Would one disallow their off topic forum? Pros? Cons?

No, I just noindex the forums (in the forum settings) I don’t want crawled and indexed. Robots.txt disallow doesn’t necessarily stop indexing.

Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.

developers.google.com

 

[mc nabulsy]

done added to my forum, hope this really help improving. is it the same with 2.2?

 

[dutchbb]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot 
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: YandexBot
User-agent: DotBot
User-agent: omgilibot
User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Barkrowler
Disallow: /


User-agent: Mediapartners-Google
Allow: /

User-agent: *
Disallow: /conversations/
Disallow: /forums/*?prefix_id
Disallow: /forums/*?order
Disallow: /forums/*?unanswered
Disallow: /forums/*?unsolved
Disallow: /threads/*?order
Disallow: /media/*?order
Disallow: /forums/*?filter_threads=
Disallow: /media/*?no_date_limit=
Disallow: /misc/language
Disallow: /misc/style
Disallow: /mailto:
Disallow: /tel:

Removed pages with noindex on them. Added all filters.

Removed nofollow on links to let google crawl because sometimes it indexes them. Nofollow (or ugc) is also propably best used only on external links.

Those 3 bots had to be added separately to work.

 

[CoinBit]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: YandexBot
User-agent: DotBot
User-agent: omgilibot
User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Barkrowler
Disallow: /


User-agent: Mediapartners-Google
Allow: /

User-agent: *
Disallow: /conversations/
Disallow: /forums/*?prefix_id
Disallow: /forums/*?order
Disallow: /forums/*?unanswered
Disallow: /forums/*?unsolved
Disallow: /threads/*?order
Disallow: /media/*?order
Disallow: /forums/*?filter_threads=
Disallow: /media/*?no_date_limit=
Disallow: /misc/language
Disallow: /misc/style
Disallow: /mailto:
Disallow: /tel:

Removed pages with noindex on them. Added all filters.

Removed nofollow on links to let google crawl because sometimes it indexes them. Nofollow (or ugc) is also propably best used only on external links.

Those 3 bots had to be added separately to work.

I have copied your code, hopefully it will continue to index my threads leaving behind the unwanted content.

Thanks