• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Xenforo.com robots.txt

dieg0

Active member
#1
Hello,

I was just checking out this site's robots file and I found this line quite interesting:

Code:
Disallow: /community/posts/
The thing is that in the forum home there are plenty of links like this one http://xenforo.com/community/posts/653990/ which Google may not be able to follow beacause of your robots.txt.

I just did a test fetching as Google on my site and the spider cannot follow that /posts/ URL.