1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Sitemap or robots.txt

Discussion in 'XenForo Questions and Support' started by bogus, Jan 11, 2011.

  1. bogus

    bogus Guest

    I was looking a bit around to disallow spiders from watching content they shouldnt.
    I got 2 things. Sitemaps and robots.txt.

    Now im not sure what to use... Do i need just one or should i use both?
    And if i use a Sitemap which one you would suggest? this or this?
    Thanks
     
  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    A sitemap helps spiders index your site.

    A robots.txt file lets you ask spiders not to index certain areas.

    But you might not need either of these things. Note that spiders are treated as guests on the forum. So if guests can't view something then spiders can't either.
     
    bogus likes this.
  3. Kaiser

    Kaiser Well-Known Member

    bogus likes this.

Share This Page