I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txtI am seeing urls with /posts/ being indexed by Google instead of the actual thread url. The /posts/ is just a redirect.
It blocks all attachment to be index.What is this page as well? http://xenforo.com/community/attachments/
It blocks all attachment to be index.
/attachments/*
Maybe you're right.I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?
Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?
Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?
I may break down and use that one as well; I was working on some custom code as part of an existing add-on I created for a different purpose will likely say 'heck with it' and use that one from Dinh.
What is the thought on including the 'Allow: /' entry? My first reaction was what this person responded with...I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txt
Take a look at xF for comparison: http://xenforo.com/robots.txt
We've had great success by checkingAllowfirst, and then checkingDisallow, the idea being thatAllowwas intended to be more specific thanDisallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this:
User-agent: * Disallow: /norobots/ Allow: /
The intent here is obvious, but thatAllow: /will cause a bot that checksAllowfirst to think it can crawl anything on the site.
Answer is it depends on the bot. Googlebot respects it following the Disallow entries above it but other bots might not and interpret it as allowing everything, essentially ignoring your explicit Disallow entries.What is the thought on including the 'Allow: /' entry?
Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.
Have you attempted the robots.txt entry?
We use essential cookies to make this site work, and optional cookies to enhance your experience.