Recommended robots.txt

[Danloona]

Hello,

I've wanted ask, what nowadays is recommended to setup robots.txt for XF2.2 installation.
Any recommendations are very appreciated, thanks!

 

[Old Nick]

Here is mine

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

 

[Danloona]

Here is mine

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

Thank you a lot!

 

[Alvin63]

Where do you put that please? Admin Panel or within the file manager? I can't see a robots.txt file in my file manager

 

[Old Nick]

FTP @ root

 

[Alvin63]

Thank you. Not sure how to do that if there isn't one there. Do I have to create it?

 

[eL_]

Thank you. Not sure how to do that if there isn't one there. Do I have to create it?

Make a new "Text Document" on your device. Open it, copy/paste what @Nicolas FR post early on. Save that document as "robot.txt" and after upload it on your XenForo root folder via FTP. Is usually named "public_html" or "www".

 

[Alvin63]

Is robot.txt without and s or robots.txt with an s?

 

[Alvin63]

Here is mine

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

If I just wanted to block SEMrush but not google and others, would it just be:

User-agent: SemrushBot
Disallow: /
Allow: /

Also is it case specific? I see SEMrushbot showing up with capitals like that

 

[TPerry]

If I just wanted to block SEMrush but not google and others, would it just be:

You need to block the lower listed ones also.. otherwise, with your history of worrying about 404 errors and whatnot in Google results, you will go nuts as you will have a LOT of "bad links".
I gave you mine in your other post.

 

[eDaddi]

Here is mine

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

After looking at what Bing has found I also added:

Disallow: /conversations/

 

[DarkGizmo]

This is what I have, perhaps mine isn't set properly?

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: ZoomSpider
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /

Also, I have a sitemap.php, not a sitemap.xml file? How would one get the XML file?

 

[bzcomputers]

Also, I have a sitemap.php, not a sitemap.xml file? How would one get the XML file?

There are the same, or I should say have the same data in them. You can use them interchangeably - just don't use both.

 

[Old Nick]

just don't use both.

What could be the problem ?

After looking at what Bing has found I also added:

I've never had an indexed conversation page, that's weird.

 

[bzcomputers]

What could be the problem ?

It can show as duplicate submissions in Google Search Console, and can cause a recurring coverage issue alert to be created.

 

[Old Nick]

I use both for years and never had this kind of alert.

 

[Alvin63]

This is what I have, perhaps mine isn't set properly?

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: ZoomSpider
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

User-agent: ltx71 - (http://ltx71.com/)
Disallow: /

Also, I have a sitemap.php, not a sitemap.xml file? How would one get the XML file?

Thanks!

 

[Old Nick]

It can show as duplicate submissions in Google Search Console, and can cause a recurring coverage issue alert to be created.

I use both for years and never had this kind of alert.

But I finally deleted one anyway, there's no point in keeping both.
I kept the .php which is updated faster than the .xml, I don't know why.

 

[Rinkrat]

How do I add robots.txt with the cloud hosting?

 

[Paul B]

Near the bottom of this page admin.php?options/groups/basicBoard/ in the ACP.