1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Xenforo.com robots.txt

Discussion in 'General XenForo Discussion and Feedback' started by dieg0, Oct 21, 2013.

  1. dieg0

    dieg0 Active Member


    I was just checking out this site's robots file and I found this line quite interesting:

    Disallow: /community/posts/
    The thing is that in the forum home there are plenty of links like this one http://xenforo.com/community/posts/653990/ which Google may not be able to follow beacause of your robots.txt.

    I just did a test fetching as Google on my site and the spider cannot follow that /posts/ URL.
  2. Brogan

    Brogan XenForo Moderator Staff Member

  3. dieg0

    dieg0 Active Member

Share This Page