Why does GWT show errors for domain.com/posts/<number> ?

[TheBigK]

My robots.txt file looks like this:

User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

...and yet; GWT reports hundreds of errors for URLs like:
http://www.domain.com/posts/300709/


Should I be worried? Is there anything that needs a fix?

 

[CTXMedia]

What is the nature of the error? 404 - 301/2 - 500 - or something else?

And have you made any changes to your robots.txt recently in relation to /posts/ or changed your site structure in any way?

 

[TheBigK]

What is the nature of the error? 404 - 301/2 - 500 - or something else?

And have you made any changes to your robots.txt recently in relation to /posts/ or changed your site structure in any way?

I did not make any changes. There is no error code associated with those errors! That's why I'm curious to know what's really happening.

I found that GWT now allows you to download a bit of your total errors. The downloaded CSV shows no error code or type against these errors.

 

[CTXMedia]

Could you provide a screenshot of the error text as it is displayed in GWT or perhaps explain what section of GWT the errors are in?

 

[TheBigK]

Could you provide a screenshot of the error text as it is displayed in GWT or perhaps explain what section of GWT the errors are in?

This isn't very useful, I guess :-



WMT shows only top 1000 errors so these errors can only be seen after downloading the error log.

 

[CTXMedia]

That isn't a screenshot - but I'm guessing you got this from Crawl > Crawl Errors ?

Anyway, it looks like GWT is reporting the redirection that is in place:

• http://www.crazyengineers.com/posts/304153/ - redirects to:
• http://www.crazyengineers.com/threads/anyone-here-fan-of-tv-series.60789/page-3#post-304153

If that's the case it's nothing to worry about - let Google resolve them itself.

 

[TheBigK]

Yep - they do redirect properly. So there should not be any issue. However, Google reporting them as error got me worried. My bigger question is: Why is Google ignoring our robots.txt.