• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

robots.txt

  1. U

    XF 1.5 About Robots

    Hello, How can I add robots/bots to my forum? E.g.: Total: 318 (members: 27, guests: 223, robots: 68)
  2. Alfa1

    Lack of interest Automatically synch robots.txt with sitemaps

    Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in robots.txt or it will cause crawl errors. Conversely: if a directory is not disallowed in robots.txt, then it should not be inaccessible...
  3. X

    Robots Disallow - Why Allow at End

    Noticed on this websites robots.txt that their disallow ended with an "Allow /" User-agent: * Disallow: /community/find-new/ Disallow: /community/account/ Disallow: /community/attachments/ Disallow: /community/goto/ Disallow: /community/posts/ Disallow: /community/login/ Disallow...
  4. Mr Lucky

    Lack of interest Option for sitemap to obey robots.txt

    I think it would be a good idea if there was an option to make the sitemap follow robots.txt rules. It would mean the sitemap more accurately follows what you actually want indexed. See this post here: by @cmeinck...