1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Automatically synch robots.txt with sitemaps

Discussion in 'XenForo Suggestions' started by Alfa1, Jul 31, 2016.

  1. Alfa1

    Alfa1 Well-Known Member

    Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in robots.txt or it will cause crawl errors.

    Conversely: if a directory is not disallowed in robots.txt, then it should not be inaccessible to guests.

    Since XenForo has sitemaps built in, it should also synch robots.txt to avoid conflicts between sitemaps and robots.txt and thereby avoid crawl errors.

    What I am proposing is not on a url level, but on a directory level. If guests have no access to node_X, then robots.txt should disallow node_X.
    Robots.txt should be rebuild every time the sitemap is rebuild.
  2. ftsservice

    ftsservice Active Member

    I respectfully disagree. Such a feature could conflict with other software which has rules in robots.txt or software packages which use a dynamically created robots.txt file to handle various crawlers.
    Claudio likes this.
  3. Alfa1

    Alfa1 Well-Known Member

    Good point, but robots.txt rules that fall outside of xenforo could be catered to with a field with additional rules.
    And of course such function should have an On/Off toggle.
  4. ftsservice

    ftsservice Active Member

    If those abilities could be added to it, then it would be workable for my situation. :)

Share This Page