Xenforo, Canonical URLs & 'Not Selected' URLs in GWT

[TheBigK]

First of all, I admit that I don't know just as much I should know about canonical URLs. All I know is that Google wants to know which URL is the best representative of the content, if there are several URLs that point to one page. [Ex. www.xenforo.com vs xenforo.com ]

While investigating the possible causes of traffic drop on our website, I found the following in Google webmaster tools -

Note that the 'Not Selected' count has grown up way beyond what it should have been. Look at the beginning of the lines though-

1. We switched from vB -> XF on December 2011. This is where the 'Not Selected' count is a little above the 'total indexed' count.

2. Sometime between April & May, the count goes up by huge margin! I believe this is the time when we installed the tag-interlinking plugin on our wordpress setup (but not sure).

I wish to know from XF webmasters -

1. Do you see similar trend in your GWT (especially those who've converted from vB or other boards) and setup 301 redirects.

2. How does Xenforo handle canonical URLs?

3. If you have 'Not Selected' more than 'Total Indexed' - what are the places to look for in Xenforo? I'm going to look at WordPress, but I want to be absolutely sure that there's nothing to worry about on the XF front.

I'll be thankful to everyone who reads and responds to this post.

 

[TheBigK]

Sorry to bump this. I see Xenforo adding 'Canonical' tag to the URL in the header. Does it take care of everything?

 

[Jake Bunce]

XenForo has a canonical URL in the head of the page. It also does an automatic 301 redirect to the canonical URL for important routes like forums and threads to ensure those pages are only accessed via the canonical URL.

 

[IanC]

I've also got a crazy number of "not selected" URLS - almost a million on a forum with just a few hundred threads...

Seems like something isn't correct, surely? There's nothing other than an XF installation on the domain.

 

[TheBigK]

I've also got a crazy number of "not selected" URLS - almost a million on a forum with just a few hundred threads...

Seems like something isn't correct, surely? There's nothing other than an XF installation on the domain.

Did you migrate from vB to xf and added 301 redirects? If yes, then that count will go up.

 

[IanC]

Yes, I migrated - but I only redirected a handful of threads. I looked at another fresh XF site and that too has the same problem with almost a million "not selected" URLs and only a small number of threads.

I did some more digging and it looks like Google may be trying to index search links - as there are loads of domain.com/search/123456 URLs being attempted.

Perhaps search links should be "nofollow" - could that be the cause? I'd be interested to see if other XF users are getting this same problem, as it's on all my XF domains.

 

[TheBigK]

The URL google reports in my case see to be of different nature, but of course I don't have access to all the URLs it has 'not selected'. Would love to have a fix for this. Blocking the /search/ won't solve the problem. We need to add noindex tag to it.

@Jake Bunce: Will the solution here - http://xenforo.com/community/threads/adding-noindex-to-some-pages.39513/#post-448323 work in this case ?

 

[Mouth]

User-agent: *
Disallow: /index.php?search/
Disallow: /search/

... in your robots.txt ??

 

[Jake Bunce]

@Jake Bunce: Will the solution here - http://xenforo.com/community/threads/adding-noindex-to-some-pages.39513/#post-448323 work in this case ?

User-agent: *
Disallow: /index.php?search/
Disallow: /search/

... in your robots.txt ??

Either one works.

 

[TheBigK]

Either one works.

Thanks. I'll go the 'noindex' way. Why is it that Google bot tries to index the /search/ threads?

 

[ibenick]

Has anyone seen any improvement on this by adding the search disallow into robots.txt? I also have a million or so of these 'Not Selected' URLs.

 

[TheBigK]

I got the 'not selected' count down by 50%, which is a big gain! Don't block content through robots.txt, apart from what you'd see in the robots.txt here on XF. Has your traffic affected?

 

[ibenick]

My traffic has not decreased, but perhaps it has not increased as fast as it should?

Curious, what is the reason for not doing it through robots.txt?

I have mine setup the same as at http://xenforo.com/robots.txt except i also added /members although that doesn't seem to have helped much as my members are still all over in search results.

 

[ibenick]

One other thing. Per the instructions from Jake in the thread listed above, I understand I should add this to search and members:

<xen:container var="$head.robots">
    <meta name="robots" content="noindex" /></xen:container>

But what exact templates do I do that in? Member_view and?

 

[Jake Bunce]

One other thing. Per the instructions from Jake in the thread listed above, I understand I should add this to search and members:

<xen:container var="$head.robots">
    <meta name="robots" content="noindex" /></xen:container>

But what exact templates do I do that in? Member_view and?

In the root templates for those pages:

http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/