Errors and robots.txt

[Robby]

Hi,

I'm getting a lot of errors since i changed robots.txt file. Members, gallery and resources get faults in webmaster tools. Can anyone tell me what is wrong with this robots.txt:

User-agent: *
User-agent: Mediapartners-Google
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Sitemap: http://site.com/sitemap.php

 

[Paul B]

What faults are you seeing in GWT?

I don't see any reference to those routes.

 

[Robby]

members: reaction code 403
resources: 403
gallery: 403

Getting this message from google:

Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.

Recommended action

• Check the Crawl Errors page in Webmaster Tools.
• Check if your server or host is blocking Googlebot.
• If you don't want Google to crawl these pages, consider adding them to your robots.txt.

Learn more in our Help Center.

 

[Paul B]

There are various tools in GWT you can use such as 'Fetch as Google' and 'robots.txt tester'.
You will need to try and determine why Google is reporting those URLs as errors.

 

[Robby]

There are various tools in GWT you can use such as 'Fetch as Google' and 'robots.txt tester'.
You will need to try and determine why Google is reporting those URLs as errors.

All works in there. Strange