1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Xenforo, Canonical URLs & 'Not Selected' URLs in GWT

Discussion in 'General XenForo Discussion and Feedback' started by TheBigK, Nov 23, 2012.

  1. TheBigK

    TheBigK Well-Known Member

    First of all, I admit that I don't know just as much I should know about canonical URLs. All I know is that Google wants to know which URL is the best representative of the content, if there are several URLs that point to one page. [Ex. www.xenforo.com vs xenforo.com ]

    While investigating the possible causes of traffic drop on our website, I found the following in Google webmaster tools -
    Note that the 'Not Selected' count has grown up way beyond what it should have been. Look at the beginning of the lines though-

    1. We switched from vB -> XF on December 2011. This is where the 'Not Selected' count is a little above the 'total indexed' count.

    2. Sometime between April & May, the count goes up by huge margin! I believe this is the time when we installed the tag-interlinking plugin on our wordpress setup (but not sure).

    I wish to know from XF webmasters -

    1. Do you see similar trend in your GWT (especially those who've converted from vB or other boards) and setup 301 redirects.

    2. How does Xenforo handle canonical URLs?

    3. If you have 'Not Selected' more than 'Total Indexed' - what are the places to look for in Xenforo? I'm going to look at WordPress, but I want to be absolutely sure that there's nothing to worry about on the XF front.

    I'll be thankful to everyone who reads and responds to this post.
  2. TheBigK

    TheBigK Well-Known Member

    Sorry to bump this. I see Xenforo adding 'Canonical' tag to the URL in the header. Does it take care of everything?
  3. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    XenForo has a canonical URL in the head of the page. It also does an automatic 301 redirect to the canonical URL for important routes like forums and threads to ensure those pages are only accessed via the canonical URL.
  4. IanC

    IanC Active Member

    I've also got a crazy number of "not selected" URLS - almost a million on a forum with just a few hundred threads...

    Seems like something isn't correct, surely? There's nothing other than an XF installation on the domain.
  5. TheBigK

    TheBigK Well-Known Member

    Did you migrate from vB to xf and added 301 redirects? If yes, then that count will go up.
  6. IanC

    IanC Active Member

    Yes, I migrated - but I only redirected a handful of threads. I looked at another fresh XF site and that too has the same problem with almost a million "not selected" URLs and only a small number of threads.

    I did some more digging and it looks like Google may be trying to index search links - as there are loads of domain.com/search/123456 URLs being attempted.

    Perhaps search links should be "nofollow" - could that be the cause? I'd be interested to see if other XF users are getting this same problem, as it's on all my XF domains.
  7. TheBigK

    TheBigK Well-Known Member

  8. Mouth

    Mouth Well-Known Member

    User-agent: *
    Disallow: /index.php?search/
    Disallow: /search/
    ... in your robots.txt ??
  9. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

  10. TheBigK

    TheBigK Well-Known Member

    Thanks. I'll go the 'noindex' way. Why is it that Google bot tries to index the /search/ threads? o_O
  11. ibenick

    ibenick Active Member

    Has anyone seen any improvement on this by adding the search disallow into robots.txt? I also have a million or so of these 'Not Selected' URLs.
  12. TheBigK

    TheBigK Well-Known Member

    I got the 'not selected' count down by 50%, which is a big gain! Don't block content through robots.txt, apart from what you'd see in the robots.txt here on XF. Has your traffic affected?
  13. ibenick

    ibenick Active Member

    My traffic has not decreased, but perhaps it has not increased as fast as it should?

    Curious, what is the reason for not doing it through robots.txt?

    I have mine setup the same as at http://xenforo.com/robots.txt except i also added /members although that doesn't seem to have helped much as my members are still all over in search results.
  14. ibenick

    ibenick Active Member

    One other thing. Per the instructions from Jake in the thread listed above, I understand I should add this to search and members:

    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    But what exact templates do I do that in? Member_view and?
  15. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

Share This Page