Robots.txt

[Dynamic]

Hi everyone,

Google sent me a message saying their bot has come across errors on my site. I would like to prevent the /reply after a thread from being crawled. How do I do this?

Eg board/threads/topictile.111/reply

I would still like it to crawl board/threads/topictitle.111 but I do not wish for it to follow the /reply page.

Thanks.

 

[Chris D]

Are you sure it's that page generating the error?

It's already set to noindex:

<meta name="robots" content="noindex" />

 

[Dynamic]

Nope, webmaster tools is telling me there is a crawl error. Where would I be able to find the template to add the meta tag in?

Thanks.

EG

board/threads/nators-tank.1142/reply
Last crawled: 7/29/12
First detected: 7/29/12
The target URL does not exist and your server is not returning a 404 (file not found) error. More info.
Your server returns a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404). This creates a poor experience for searchers and search engines.
More information about "soft 404" errors

 

[Chris D]

You should see it's already in the thread_reply template.

Maybe tackling this as well, additionally with robots.txt might help:


User-Agent: *
Disallow: board/threads/nators-tank.1142/reply


Those two lines in your robots.txt file might help, though, again, it's already doing that.


I was curious, though, so I searched and found your board. You seem to have guest posting switched on so, technically, Google shouldn't be getting a soft 404 error, regardless of what's in the meta or robots.txt file.

I do notice, though, that the error is from nearly a month ago. If you've switched on guest posting since that error, then chances are Google won't be experiencing those errors anymore.

Other possibilities for those errors include temporary server issues, or maybe the thread was deleted or closed temporarily or some other explanation.

It's not normally too much to worry about.

 

[Dynamic]

Hi Chris,

Thanks for the reply. The problem is there is another 147 pages with the same error... It is not that single page.

 

[Chris D]

And the date of the errors?

Are they /reply pages too, or different?

 

[Dynamic]

All /reply pages. From about a month ago, none recently.

 

[Chris D]

When did you enable Guest posting?

 

[MagnusB]

It sounds like the link is displaying to guest. Besides, the default URL for xF is /add-reply, not /reply.

I would visit the threads as a guest, and see if you can find the URL for it, follow it and see what it does.