XF 2.2 Recommended robots.txt

Danloona

Member
Hello,

I've wanted ask, what nowadays is recommended to setup robots.txt for XF2.2 installation.
Any recommendations are very appreciated, thanks!
 
Solution
Here is mine
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml
Here is mine
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml
 
Solution
Here is mine
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml
Thank you a lot!
 
Here is mine
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml
If I just wanted to block SEMrush but not google and others, would it just be:

User-agent: SemrushBot
Disallow: /
Allow: /

Also is it case specific? I see SEMrushbot showing up with capitals like that
 
If I just wanted to block SEMrush but not google and others, would it just be:
You need to block the lower listed ones also.. otherwise, with your history of worrying about 404 errors and whatnot in Google results, you will go nuts as you will have a LOT of "bad links".
I gave you mine in your other post.
 
Here is mine
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml
After looking at what Bing has found I also added:
Disallow: /conversations/
 
This is what I have, perhaps mine isn't set properly?

Code:
User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: ZoomSpider
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /


Also, I have a sitemap.php, not a sitemap.xml file? How would one get the XML file?
 
This is what I have, perhaps mine isn't set properly?

Code:
User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: ZoomSpider
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /


Also, I have a sitemap.php, not a sitemap.xml file? How would one get the XML file?
Thanks!
 
Top Bottom