• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Sitemap or robots.txt

B

bogus

Guest
#1
I was looking a bit around to disallow spiders from watching content they shouldnt.
I got 2 things. Sitemaps and robots.txt.

Now im not sure what to use... Do i need just one or should i use both?
And if i use a Sitemap which one you would suggest? this or this?
Thanks
 

Jake Bunce

XenForo moderator
Staff member
#2
A sitemap helps spiders index your site.

A robots.txt file lets you ask spiders not to index certain areas.

But you might not need either of these things. Note that spiders are treated as guests on the forum. So if guests can't view something then spiders can't either.