1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Lack of Interest Option to exclude selected forums from sitemap

Discussion in 'Closed Suggestions' started by DJ XtAzY, Jul 22, 2014.

  1. DJ XtAzY

    DJ XtAzY Active Member

    For the site map generator, can we have an option to determine whether or not we want to include threads created in that particular node/forum to be part of the sitemap. The option would most likely be in the Edit Forum or Create New Forum.
     
    Optic, cdub and Mr Lucky like this.
  2. Mike

    Mike XenForo Developer Staff Member

    Presumably you actually want an option to automatically "noindex" threads in a forum, as simply not submitting them in the sitemap won't really do anything.

    The purpose of a sitemap is to aid in discovery of content; it doesn't stop crawling of your site.
     
  3. Hoffi

    Hoffi Well-Known Member

    The only logic switch for this, ist the right for guests. Becazse Google see everything what guests can view.

    If didn't want indexed dome forums, close them for guests.

    If set the no-index meta tag, or disallow it via robots.txt, not all crawler will really react to this setting. There are several - very agressive - crawler that crawls everything they find.
     
  4. Karelke

    Karelke Active Member

    You can easily exclude a node in the robots.txt file.
     
  5. Jeremy

    Jeremy XenForo Moderator Staff Member

    A site map is not a robots.txt file and they perform entirely separate functionalities.
     
  6. cmeinck

    cmeinck Well-Known Member

    Plus one on being able to remove forum nodes from the created sitemap. For example, I no-index my Off Topic threads. I let Google crawl them, but don't want them indexed. If I was able to remove them from my sitemap, it would provide a more accurate view of indexation on GWT. I'm submitting thousands of URLs on my sitemap that aren't indexed. I'd much rather have a tight sitemap that reflected actual indexation of threads that I want indexed.
     
  7. Mr Lucky

    Mr Lucky Well-Known Member

    I would like this also.
     
    cdub likes this.
  8. Kintaro

    Kintaro Well-Known Member

    with latest xenforo versions is now possibile excludes nodes from the sitemap?

    ...or as I suspect set no-index in a specific node is enough for Google's crawlers?
     
  9. XenBurger

    XenBurger Member

    Looks like I'm plus-one'ing this as well.

    There are a lot of side effects of having something in a sitemap that shouldn't be seen by google.
     
  10. Optic

    Optic Well-Known Member

  11. XenBurger

    XenBurger Member

    1) There is an AddOn or two which allow you to noindex certain forums. This is definitely the best approach.

    2) However, Google is a naggy naggy little creature, and if it sees *anything* in your sitemap which its not allowed to index, or doesn't exist, you quickly end up with tens/hundreds of thousands of "Crawl Errors" in that portion of the site, and they will pile up very quickly.

    While 404's and 410's don't harm your rankings, they will stay there for months, drowning out your ability to see actual problem URLs. Google only allows you to clear out 1,000 of them per day, and in my case (300,000 of them due to different causes) I am certain that I'm missing some important pages that shouldn't be 404ing elsewhere on my site that have nothing to do with XenForo.

    If the sitemap contains something Google isn't allowed to see, or can't reach (410, 404, or even 403-forbidden) it will list the Sitemap as the cause for the generation of the error. So it seems like a good idea that the sitemap never shows threads which it can't readily fetch.

    I may be wrong on some of this.
     

Share This Page