Robots.txt again now for 2.3.5

[Seeker-Smith]

Is there a default robot.txt that we should be using?

 

[Kirby]

No.

 

[Seeker-Smith]

No.

Is there a best practice in creating one?

 

[bzcomputers]

Is there a best practice in creating one?

Very basic, just disallow all areas of your site that do not have content relevant to index, or content that is not unique (already found elsewhere on your site). This in turn improves SEO and saves crawler traffic for the important stuff.

For a stock XenForo site this could be something like:

Sitemap: https://www.YourDomain.com/community/sitemap.xml

User-agent: *
Disallow: /community/admin.php
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/direct-messages/
Disallow: /community/goto/
Disallow: /community/login/
Disallow: /community/lost-password/
Disallow: /community/online/
Disallow: /community/posts/
Disallow: /community/register/
Disallow: /community/search/
Disallow: /community/whats-new/

If you don't want member pages indexed add:

Disallow: /community/members/

If you aren't using tags add:

Disallow: /community/tags/

If you're using Google Adsense add this just below your sitemap reference:

User-agent: Mediapartners-Google
Disallow:

 

[Seeker-Smith]

Very basic, just disallow all areas of your site that do not have content relevant to index, or content that is not unique (already found elsewhere on your site). This in turn improves SEO and saves crawler traffic for the important stuff.

For a stock XenForo site this could be something like:

Sitemap: https://www.YourDomain.com/community/sitemap.xml

User-agent: *
Disallow: /community/admin.php
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/direct-messages/
Disallow: /community/goto/
Disallow: /community/login/
Disallow: /community/lost-password/
Disallow: /community/online/
Disallow: /community/posts/
Disallow: /community/register/
Disallow: /community/search/
Disallow: /community/whats-new/

If you don't want member pages indexed add:

Disallow: /community/members/

If you aren't using tags add:

Disallow: /community/tags/

If you're using Google Adsense add this just below your sitemap reference:

User-agent: Mediapartners-Google
Disallow:

Thanks.

Is there anything special to do if you have XFMG?

 

[bzcomputers]

Thanks.

Is there anything special to do if you have XFMG?

I personally don't make any additional changes to robots.txt for sites with XFMG or XFRM.

 

[Seeker-Smith]

Thank you again!