XF 1.1 /posts/ urls being indexed by Google?

[Brent W]

I am seeing urls with /posts/ being indexed by Google instead of the actual thread url. The /posts/ is just a redirect.

See attachment

 

[AlexT]

I am seeing urls with /posts/ being indexed by Google instead of the actual thread url. The /posts/ is just a redirect.

I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txt

Take a look at xF for comparison: http://xenforo.com/robots.txt

 

[Brent W]

Hmm... how is the redirect handled for /posts/? It's obviously a redirect so why is Google indexing that url instead of /threads/?

 

[Brent W]

What is this page as well? http://xenforo.com/community/attachments/

 

[rdn]

What is this page as well? http://xenforo.com/community/attachments/

It blocks all attachment to be index.
/attachments/*

 

[Brent W]

It blocks all attachment to be index.
/attachments/*

I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?

 

[rdn]

I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?

Maybe you're right.

 

[Brent W]

This is quite the flaw is it not for SEO? My Google Webmaster tools sitemap was not showing the /treads/ urls as being indexed because it was actually indexing /posts/ which is a redirect.

I'm still confused why Google was indexing those urls when it takes you to a page that uses rel="canonical"

 

[rdn]

Anyway, here's mine: http://www.phcorner.net/robots.txt

 

[Kevin]

I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?

Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?

 

[Brent W]

Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?

I use this: http://xenforo.com/community/resources/tinhte-image-attachment-optimization-cdn-support.909/

 

[Kevin]

I use this: http://xenforo.com/community/resources/tinhte-image-attachment-optimization-cdn-support.909/

I may break down and use that one as well; I was working on some custom code as part of an existing add-on I created for a different purpose will likely say 'heck with it' and use that one from Dinh.


(Chris, if you are reading this, you may have a chuckle at my expense. )

 

[Jeremy]

For the record, I just did a quick search and network analysis on /posts/. They are correctly giving a 301 Redirect (Permanent) which is the suggested practice by Google:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93633

 

[Kevin]

I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txt

Take a look at xF for comparison: http://xenforo.com/robots.txt

What is the thought on including the 'Allow: /' entry? My first reaction was what this person responded with...

http://webmasters.stackexchange.com...ay-to-handle-allow-and-disallow-in-robots-txt

We've had great success by checkingAllowfirst, and then checkingDisallow, the idea being thatAllowwas intended to be more specific thanDisallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this:
User-agent: * Disallow: /norobots/ Allow: /
The intent here is obvious, but thatAllow: /will cause a bot that checksAllowfirst to think it can crawl anything on the site.

 

[Kevin]

What is the thought on including the 'Allow: /' entry?

Answer is it depends on the bot. Googlebot respects it following the Disallow entries above it but other bots might not and interpret it as allowing everything, essentially ignoring your explicit Disallow entries.

 

[Brent W]

This was fixed in 1.1.5 it seems. Thanks!

 

[Brent W]

I still see this happening. I don't use a robots.txt to block /posts/ as it shouldn't be necessary with how xenForo works for /posts/ right?



This is a forum that went from vBulletin to xenForo 1.2.2. The last I saw fixes to prevent /posts/ from showing up happened in 1.1.5.

Anyone else have any insight on why Google is still indexing /posts/ with what exists in the xenForo code base?

 

[Jeremy]

Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.

Have you attempted the robots.txt entry?

 

[Brent W]

Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.

Have you attempted the robots.txt entry?

I used to use it but if xenForo is using a correct 301 and it is still being indexed then something else needs to be figured out here. Google isn't perfect and Matt Cutts is available to talk to here. Something isn't right. These urls are always 301 redirected correct? There is no circumstance where these urls exist where there is not a 301 tagged to them? Therefore, they should not be indexed at all and the correct url, with the canonical tag on it, should be indexed.

 

[Jeremy]

No, there is no circumstance where a /posts link is not given a proper 301. If Google ignores the redirect and indexes it improperly, there's not much XenForo can do except redirect when it is hit. I also believe any /posts/ link is nofollow'd by default.