Sitemap or robots.txt



I was looking a bit around to disallow spiders from watching content they shouldnt.
I got 2 things. Sitemaps and robots.txt.

Now im not sure what to use... Do i need just one or should i use both?
And if i use a Sitemap which one you would suggest? this or this?

Jake Bunce

XenForo moderator
Staff member
A sitemap helps spiders index your site.

A robots.txt file lets you ask spiders not to index certain areas.

But you might not need either of these things. Note that spiders are treated as guests on the forum. So if guests can't view something then spiders can't either.