Lack of interest Option to exclude selected forums from sitemap

[DJ XtAzY]

For the site map generator, can we have an option to determine whether or not we want to include threads created in that particular node/forum to be part of the sitemap. The option would most likely be in the Edit Forum or Create New Forum.

 

[Mike]

Presumably you actually want an option to automatically "noindex" threads in a forum, as simply not submitting them in the sitemap won't really do anything.

The purpose of a sitemap is to aid in discovery of content; it doesn't stop crawling of your site.

 

[Hoffi]

The only logic switch for this, ist the right for guests. Becazse Google see everything what guests can view.

If didn't want indexed dome forums, close them for guests.

If set the no-index meta tag, or disallow it via robots.txt, not all crawler will really react to this setting. There are several - very agressive - crawler that crawls everything they find.

 

[Karelke]

You can easily exclude a node in the robots.txt file.

 

[Jeremy]

A site map is not a robots.txt file and they perform entirely separate functionalities.

 

[cmeinck]

Plus one on being able to remove forum nodes from the created sitemap. For example, I no-index my Off Topic threads. I let Google crawl them, but don't want them indexed. If I was able to remove them from my sitemap, it would provide a more accurate view of indexation on GWT. I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.

 

[Mr Lucky]

I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.

I would like this also.

 

[Kintaro]

with latest xenforo versions is now possibile excludes nodes from the sitemap?

...or as I suspect set no-index in a specific node is enough for Google's crawlers?

 

[XenBurger]

Plus one on being able to remove forum nodes from the created sitemap. For example, I no-index my Off Topic threads. I let Google crawl them, but don't want them indexed. If I was able to remove them from my sitemap, it would provide a more accurate view of indexation on GWT. I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.

Looks like I'm plus-one'ing this as well.

There are a lot of side effects of having something in a sitemap that shouldn't be seen by google.

 

[Optic]

Would be nice to see. e.g. to exclude an Introductions node.

Seems there is an add-on that could achieve this in the interim:
https://xenforo.com/community/threads/th-remove-from-sitemap.88588/

 

[XenBurger]

Presumably you actually want an option to automatically "noindex" threads in a forum, as simply not submitting them in the sitemap won't really do anything.

The purpose of a sitemap is to aid in discovery of content; it doesn't stop crawling of your site.

1) There is an AddOn or two which allow you to noindex certain forums. This is definitely the best approach.

2) However, Google is a naggy naggy little creature, and if it sees *anything* in your sitemap which its not allowed to index, or doesn't exist, you quickly end up with tens/hundreds of thousands of "Crawl Errors" in that portion of the site, and they will pile up very quickly.

While 404's and 410's don't harm your rankings, they will stay there for months, drowning out your ability to see actual problem URLs. Google only allows you to clear out 1,000 of them per day, and in my case (300,000 of them due to different causes) I am certain that I'm missing some important pages that shouldn't be 404ing elsewhere on my site that have nothing to do with XenForo.

If the sitemap contains something Google isn't allowed to see, or can't reach (410, 404, or even 403-forbidden) it will list the Sitemap as the cause for the generation of the error. So it seems like a good idea that the sitemap never shows threads which it can't readily fetch.

I may be wrong on some of this.