• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Robots.txt

Dynamic

Well-known member
#1
Hi everyone,

Google sent me a message saying their bot has come across errors on my site. I would like to prevent the /reply after a thread from being crawled. How do I do this?

Eg board/threads/topictile.111/reply

I would still like it to crawl board/threads/topictitle.111 but I do not wish for it to follow the /reply page.

Thanks.
 

Dynamic

Well-known member
#3
Nope, webmaster tools is telling me there is a crawl error. Where would I be able to find the template to add the meta tag in?

Thanks.

EG

board/threads/nators-tank.1142/reply
Last crawled: 7/29/12
First detected: 7/29/12
The target URL does not exist and your server is not returning a 404 (file not found) error. More info.
Your server returns a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404). This creates a poor experience for searchers and search engines.
More information about "soft 404" errors
 

Chris D

XenForo developer
Staff member
#4
You should see it's already in the thread_reply template.

Maybe tackling this as well, additionally with robots.txt might help:


User-Agent: *
Disallow: board/threads/nators-tank.1142/reply


Those two lines in your robots.txt file might help, though, again, it's already doing that.


I was curious, though, so I searched and found your board. You seem to have guest posting switched on so, technically, Google shouldn't be getting a soft 404 error, regardless of what's in the meta or robots.txt file.

I do notice, though, that the error is from nearly a month ago. If you've switched on guest posting since that error, then chances are Google won't be experiencing those errors anymore.

Other possibilities for those errors include temporary server issues, or maybe the thread was deleted or closed temporarily or some other explanation.

It's not normally too much to worry about.
 

Dynamic

Well-known member
#5
Hi Chris,

Thanks for the reply. The problem is there is another 147 pages with the same error... It is not that single page.
 

MagnusB

Well-known member
#9
It sounds like the link is displaying to guest. Besides, the default URL for xF is /add-reply, not /reply.

I would visit the threads as a guest, and see if you can find the URL for it, follow it and see what it does.