Xenforo.com robots.txt

[dieg0]

Hello,

I was just checking out this site's robots file and I found this line quite interesting:

Disallow: /community/posts/

The thing is that in the forum home there are plenty of links like this one http://xenforo.com/community/posts/653990/ which Google may not be able to follow beacause of your robots.txt.

I just did a test fetching as Google on my site and the spider cannot follow that /posts/ URL.

 

[Paul B]

http://xenforo.com/community/threads/why-disallow-posts-in-robots-txt.56006/#post-596780

 

[dieg0]

http://xenforo.com/community/threads/why-disallow-posts-in-robots-txt.56006/#post-596780

I guess it makes sense for you. I just recently moved from vB and I need Google to follow those redirects.

Thanks!