XF 2.0 Changes to robots.txt from Xenforo 1.5 to 2.0

[JackieChun]

Are there changes to XenForo's URL structure in the 2.0 version that warrant taking another look at your old robots.txt file? Here's what my old robots.txt looks like. I believe it was originally authored by @Brogan.

User-agent: *
Disallow: /forum/attachments/
Disallow: /forum/forums/*/?direction
Disallow: /forum/forums/*/?order
Disallow: /forum/help/
Disallow: /forum/login/
Disallow: /forum/lost-password/
Disallow: /forum/members/
Disallow: /forum/misc/
Disallow: /forum/online/
Disallow: /forum/posts/
Disallow: /forum/register/
Disallow: /forum/search/

 

[JackieChun]

anyone?

 

[SyTry]

Hello,

XenForo Community use this one :

User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

Sitemap: https://xenforo.com/community/sitemap.php

Link : https://xenforo.com/robots.txt

Regards, SyTry

 

[Ozzy47]

Hello,

XenForo Community use this one :

User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

Sitemap: https://xenforo.com/community/sitemap.php

Link : https://xenforo.com/robots.txt

Regards, SyTry

That assumes the files are in the /community folder, not in the /forum folder.

As for the OP sure, I believe what you already have should be fine.

 

[akok]

Is in xenforo 2
/find-new/ => /whats-new/

 

[dethfire]

What's the purpose for blocking /whats-new/ ?

 

[djbaxter]

What's the purpose for blocking /whats-new/ ?

Duplicate URLs.

 

[dethfire]

Duplicate URLs.

Aren't there canonicals?

 

[djbaxter]

Different issue. You don't want the bots crawling duplicate content because it's a waste of crawl budget.