Sitemap or robots.txt

  • Thread starter Thread starter bogus
  • Start date Start date
B

bogus

Guest
I was looking a bit around to disallow spiders from watching content they shouldnt.
I got 2 things. Sitemaps and robots.txt.

Now im not sure what to use... Do i need just one or should i use both?
And if i use a Sitemap which one you would suggest? this or this?
Thanks
 
A sitemap helps spiders index your site.

A robots.txt file lets you ask spiders not to index certain areas.

But you might not need either of these things. Note that spiders are treated as guests on the forum. So if guests can't view something then spiders can't either.
 
Top Bottom