1. scskmg

    Setting customize robots.txt

    google reports the following error back to me : Blocked by robots.txt file Excluded by "noindex" tag In the meantime I have set the most important directories to "allow". I am now not quite sure if I have forgotten something ..or am I on a completely wrong path? Thanks for a possible...
  2. A

    XF 1.3 ROBOTS

    Hi... My website is showing over 170 robots online due to which i think server cpu usage reaches high whoch then stops website from opening up and gives error saying the website cannot be displayed, database error. I have used a plugin in called robots, configured it but its still showing...
  3. smozgur

    XF 2.1 Post quote URLs

    When a user quoted a post, the link that leads to the quoted post is using the following URL: This is causing "Indexed, though blocked by robots.txt" errors in Google Search Console since we block /community/goto/ route in robots.txt as it is used in XenForo website robots.txt file. Isn't...
  4. A

    XF 2.0 Xenforo goto

    Hello. Can someone explain to me what goto is and is it worth adding to robots.txt?
  5. Albanian

    XF 2.1 [Google Webmasters/Console] - Warnings blocked by robots.txt

    Hi, If i look in my Google Webmaster Console i am getting 66 warnings because there are some urls indexed by Google but blocked by robots.txt The urls are almost the same and are like :
  6. Ulas K

    XF 1.5 About Robots

    Hello, How can I add robots/bots to my forum? E.g.: Total: 318 (members: 27, guests: 223, robots: 68)
  7. Alpha1

    Lack of interest Automatically synch robots.txt with sitemaps

    Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in robots.txt or it will cause crawl errors. Conversely: if a directory is not disallowed in robots.txt, then it should not be inaccessible...
  8. X

    Robots Disallow - Why Allow at End

    Noticed on this websites robots.txt that their disallow ended with an "Allow /" User-agent: * Disallow: /community/find-new/ Disallow: /community/account/ Disallow: /community/attachments/ Disallow: /community/goto/ Disallow: /community/posts/ Disallow: /community/login/ Disallow...
  9. Mr Lucky

    Lack of interest Option for sitemap to obey robots.txt

    I think it would be a good idea if there was an option to make the sitemap follow robots.txt rules. It would mean the sitemap more accurately follows what you actually want indexed. See this post here: by @cmeinck...