• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Errors and robots.txt

Robby

Well-known member
#1
Hi,

I'm getting a lot of errors since i changed robots.txt file. Members, gallery and resources get faults in webmaster tools. Can anyone tell me what is wrong with this robots.txt:

Code:
User-agent: *
User-agent: Mediapartners-Google
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Sitemap: http://site.com/sitemap.php
 

Robby

Well-known member
#3
members: reaction code 403
resources: 403
gallery: 403

Getting this message from google:

Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.

Recommended action

  • Check the Crawl Errors page in Webmaster Tools.
  • Check if your server or host is blocking Googlebot.
  • If you don't want Google to crawl these pages, consider adding them to your robots.txt.
Learn more in our Help Center.
 

Brogan

XenForo moderator
Staff member
#4
There are various tools in GWT you can use such as 'Fetch as Google' and 'robots.txt tester'.
You will need to try and determine why Google is reporting those URLs as errors.