• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Lack of interest Option to exclude selected forums from sitemap

Mike

XenForo developer
Staff member
#2
Presumably you actually want an option to automatically "noindex" threads in a forum, as simply not submitting them in the sitemap won't really do anything.

The purpose of a sitemap is to aid in discovery of content; it doesn't stop crawling of your site.
 

Hoffi

Well-known member
#3
The only logic switch for this, ist the right for guests. Becazse Google see everything what guests can view.

If didn't want indexed dome forums, close them for guests.

If set the no-index meta tag, or disallow it via robots.txt, not all crawler will really react to this setting. There are several - very agressive - crawler that crawls everything they find.
 

cmeinck

Well-known member
#6
Plus one on being able to remove forum nodes from the created sitemap. For example, I no-index my Off Topic threads. I let Google crawl them, but don't want them indexed. If I was able to remove them from my sitemap, it would provide a more accurate view of indexation on GWT. I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.
 

Kintaro

Well-known member
#8
with latest xenforo versions is now possibile excludes nodes from the sitemap?

...or as I suspect set no-index in a specific node is enough for Google's crawlers?
 
#9
Plus one on being able to remove forum nodes from the created sitemap. For example, I no-index my Off Topic threads. I let Google crawl them, but don't want them indexed. If I was able to remove them from my sitemap, it would provide a more accurate view of indexation on GWT. I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.
Looks like I'm plus-one'ing this as well.

There are a lot of side effects of having something in a sitemap that shouldn't be seen by google.
 
#11
Presumably you actually want an option to automatically "noindex" threads in a forum, as simply not submitting them in the sitemap won't really do anything.

The purpose of a sitemap is to aid in discovery of content; it doesn't stop crawling of your site.
1) There is an AddOn or two which allow you to noindex certain forums. This is definitely the best approach.

2) However, Google is a naggy naggy little creature, and if it sees *anything* in your sitemap which its not allowed to index, or doesn't exist, you quickly end up with tens/hundreds of thousands of "Crawl Errors" in that portion of the site, and they will pile up very quickly.

While 404's and 410's don't harm your rankings, they will stay there for months, drowning out your ability to see actual problem URLs. Google only allows you to clear out 1,000 of them per day, and in my case (300,000 of them due to different causes) I am certain that I'm missing some important pages that shouldn't be 404ing elsewhere on my site that have nothing to do with XenForo.

If the sitemap contains something Google isn't allowed to see, or can't reach (410, 404, or even 403-forbidden) it will list the Sitemap as the cause for the generation of the error. So it seems like a good idea that the sitemap never shows threads which it can't readily fetch.

I may be wrong on some of this.