1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt

Discussion in 'Troubleshooting and Problems' started by Dynamic, Aug 23, 2012.

  1. Dynamic

    Dynamic Well-Known Member

    Hi everyone,

    Google sent me a message saying their bot has come across errors on my site. I would like to prevent the /reply after a thread from being crawled. How do I do this?

    Eg board/threads/topictile.111/reply

    I would still like it to crawl board/threads/topictitle.111 but I do not wish for it to follow the /reply page.

    Thanks.
     
  2. Chris D

    Chris D XenForo Developer Staff Member

    Are you sure it's that page generating the error?

    It's already set to noindex:

    HTML:
    <meta name="robots" content="noindex" />
     
    Jake Bunce likes this.
  3. Dynamic

    Dynamic Well-Known Member

    Nope, webmaster tools is telling me there is a crawl error. Where would I be able to find the template to add the meta tag in?

    Thanks.

    EG

    board/threads/nators-tank.1142/reply
    Last crawled: 7/29/12
    First detected: 7/29/12
    The target URL does not exist and your server is not returning a 404 (file not found) error. More info.
    Your server returns a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404). This creates a poor experience for searchers and search engines.
    More information about "soft 404" errors
     
  4. Chris D

    Chris D XenForo Developer Staff Member

    You should see it's already in the thread_reply template.

    Maybe tackling this as well, additionally with robots.txt might help:


    User-Agent: *
    Disallow: board/threads/nators-tank.1142/reply


    Those two lines in your robots.txt file might help, though, again, it's already doing that.


    I was curious, though, so I searched and found your board. You seem to have guest posting switched on so, technically, Google shouldn't be getting a soft 404 error, regardless of what's in the meta or robots.txt file.

    I do notice, though, that the error is from nearly a month ago. If you've switched on guest posting since that error, then chances are Google won't be experiencing those errors anymore.

    Other possibilities for those errors include temporary server issues, or maybe the thread was deleted or closed temporarily or some other explanation.

    It's not normally too much to worry about.
     
  5. Dynamic

    Dynamic Well-Known Member

    Hi Chris,

    Thanks for the reply. The problem is there is another 147 pages with the same error... It is not that single page.
     
  6. Chris D

    Chris D XenForo Developer Staff Member

    And the date of the errors?

    Are they /reply pages too, or different?
     
  7. Dynamic

    Dynamic Well-Known Member

    All /reply pages. From about a month ago, none recently.
     
  8. Chris D

    Chris D XenForo Developer Staff Member

    When did you enable Guest posting?
     
  9. MagnusB

    MagnusB Well-Known Member

    It sounds like the link is displaying to guest. Besides, the default URL for xF is /add-reply, not /reply.

    I would visit the threads as a guest, and see if you can find the URL for it, follow it and see what it does.
     

Share This Page