• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.1 /posts/ urls being indexed by Google?

Brent W

Well-known member
#1
I am seeing urls with /posts/ being indexed by Google instead of the actual thread url. The /posts/ is just a redirect.

See attachment

Screen Shot 2013-04-16 at 12.50.54 PM.png
 

Brent W

Well-known member
#3
Hmm... how is the redirect handled for /posts/? It's obviously a redirect so why is Google indexing that url instead of /threads/?
 

Brent W

Well-known member
#8
This is quite the flaw is it not for SEO? My Google Webmaster tools sitemap was not showing the /treads/ urls as being indexed because it was actually indexing /posts/ which is a redirect.

I'm still confused why Google was indexing those urls when it takes you to a page that uses rel="canonical"
 

Kevin

Well-known member
#10
I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?
Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?
 

Kevin

Well-known member
#14
I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txt

Take a look at xF for comparison: http://xenforo.com/robots.txt
What is the thought on including the 'Allow: /' entry? My first reaction was what this person responded with...

http://webmasters.stackexchange.com...ay-to-handle-allow-and-disallow-in-robots-txt

We've had great success by checkingAllowfirst, and then checkingDisallow, the idea being thatAllowwas intended to be more specific thanDisallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this:
User-agent: * Disallow: /norobots/ Allow: /
The intent here is obvious, but thatAllow: /will cause a bot that checksAllowfirst to think it can crawl anything on the site.
 

Kevin

Well-known member
#15
What is the thought on including the 'Allow: /' entry?
Answer is it depends on the bot. Googlebot respects it following the Disallow entries above it but other bots might not and interpret it as allowing everything, essentially ignoring your explicit Disallow entries.
 

Brent W

Well-known member
#17
I still see this happening. I don't use a robots.txt to block /posts/ as it shouldn't be necessary with how xenForo works for /posts/ right?

Screenshot 2013-10-28 21.23.04.png

This is a forum that went from vBulletin to xenForo 1.2.2. The last I saw fixes to prevent /posts/ from showing up happened in 1.1.5.

Anyone else have any insight on why Google is still indexing /posts/ with what exists in the xenForo code base?
 

Jeremy

Well-known member
#18
Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.

Have you attempted the robots.txt entry?
 

Brent W

Well-known member
#19
Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.

Have you attempted the robots.txt entry?
I used to use it but if xenForo is using a correct 301 and it is still being indexed then something else needs to be figured out here. Google isn't perfect and Matt Cutts is available to talk to here. Something isn't right. These urls are always 301 redirected correct? There is no circumstance where these urls exist where there is not a 301 tagged to them? Therefore, they should not be indexed at all and the correct url, with the canonical tag on it, should be indexed.
 

Jeremy

Well-known member
#20
No, there is no circumstance where a /posts link is not given a proper 301. If Google ignores the redirect and indexes it improperly, there's not much XenForo can do except redirect when it is hit. I also believe any /posts/ link is nofollow'd by default.