1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Errors and robots.txt

Discussion in 'XenForo Questions and Support' started by Robby, Apr 22, 2015.

  1. Robby

    Robby Well-Known Member


    I'm getting a lot of errors since i changed robots.txt file. Members, gallery and resources get faults in webmaster tools. Can anyone tell me what is wrong with this robots.txt:

    User-agent: *
    User-agent: Mediapartners-Google
    Disallow: /find-new/
    Disallow: /account/
    Disallow: /attachments/
    Disallow: /goto/
    Disallow: /posts/
    Disallow: /login/
    Disallow: /admin.php
    Allow: /
    Sitemap: http://site.com/sitemap.php
  2. Brogan

    Brogan XenForo Moderator Staff Member

    What faults are you seeing in GWT?

    I don't see any reference to those routes.
  3. Robby

    Robby Well-Known Member

    members: reaction code 403
    resources: 403
    gallery: 403

    Getting this message from google:

    Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.

    Recommended action

    • Check the Crawl Errors page in Webmaster Tools.
    • Check if your server or host is blocking Googlebot.
    • If you don't want Google to crawl these pages, consider adding them to your robots.txt.
    Learn more in our Help Center.
  4. Brogan

    Brogan XenForo Moderator Staff Member

    There are various tools in GWT you can use such as 'Fetch as Google' and 'robots.txt tester'.
    You will need to try and determine why Google is reporting those URLs as errors.
  5. Robby

    Robby Well-Known Member

    All works in there. Strange

Share This Page