Recommended robots.txt for XF 2.3

[frm]

I found this, and used a modified version of it for XF 2.2.

However, I was wondering if any changes should be made for XF 2.3

Here is mine

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

 

[Old Nick]

Good question. Maybe we can add this

Disallow: /featured/

 

[frm]

Good question. Maybe we can add this

Disallow: /featured/

I added it simply because I don't use that feature.

However, would the thought behind this be that featured items are just duplicate links showing up elsewhere too?

 

[Old Nick]

I thought about this page because the content is changing so I don't know if it couldn't create 404 errors.

We need expert advices about this…

 

[Anatoliy]

not an expert, but I don't think that changing content is bad. content on home pages is changing, and that is good.

 

[Old Nick]

not an expert, but I don't think that changing content is bad. content on home pages is changing, and that is good.

/whats-new/ is blocked by robots.txt due to his changing content I guess…

 

[PhineasD]

Anyone disallow the /post- pages?

 

[Old Nick]

Anyone disallow the /post- pages?

/post- and /post/ the same?

https://xenforo.com/robots.txt

 

[PhineasD]

/post- and /post/ the same?

https://xenforo.com/robots.txt

No, I mean this type of page:

https://xenforo.com/community/threads/approving-moderation-queue-twice.225539/#post-1708536
/#post-

Not this other one, which is the one that appears in the XF robots:

https://xenforo.com/community/whats-new/posts/3166031/
/posts/

 

[frm]

No. /post-NNNN takes you to the specific post in a thread.



Your #8 post is really:
https://xenforo.com/community/threads/recommended-robots-txt-for-xf-2-3.225039/post-1708535

Since https://xenforo.com/community/threads/recommended-robots-txt-for-xf-2-3.225039/ is already crawled, do you need to block crawling of /post-NNNN as it's just a link to a post on any one of the pages that it's on?

 

[akok]

After studying the errors in the webmaster's office and viewing the search results, I settled on this version of the robots file.. It was necessary to take into account some features of search engines.

# robots.txt
User-agent: Bytespider
User-agent: ClaudeBot
Disallow: /

User-agent: Yandex
# Параметры для игнорирования: team, order, direction, do, dlfileid, act, do, id, prefix_id
Clean-param: team&order&direction&do&dlfileid&act&do&id&prefix_id
Disallow: */filters$
Disallow: */reactions$
Disallow: */preview$
Disallow: */review$
Disallow: */insert-gif$
Disallow: */post-thread*
Disallow: */download
Disallow: /login/
Disallow: /admin.php
Disallow: /whats-new/
Disallow: /misc/
Disallow: /attachments/
Disallow: /lost-password/
Disallow: /account-confirmation/
Disallow: /members/
Disallow: */reply
Disallow: /*/rate
Disallow: /*?rating=
Disallow: /*?last_days=

User-agent: *
Disallow: /login/
Disallow: /admin.php
Disallow: /whats-new/
Disallow: /misc/
Disallow: /lost-password/
Disallow: /attachments/
Disallow: */post-thread*
Disallow: */reactions$
Disallow: /*/rate
Disallow: */reply
Disallow: /members/