1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 /posts/ urls being indexed by Google?

Discussion in 'Troubleshooting and Problems' started by BamaStangGuy, Apr 16, 2013.

  1. BamaStangGuy

    BamaStangGuy Well-Known Member

    I am seeing urls with /posts/ being indexed by Google instead of the actual thread url. The /posts/ is just a redirect.

    See attachment

    Screen Shot 2013-04-16 at 12.50.54 PM.png
     
  2. AlexT

    AlexT Well-Known Member

    I recommend using a robots.txt in your site root to prevent Google from indexing those URLs. It seems you are not using any robots.txt rules right now: http://www.sportgfx.com/robots.txt

    Take a look at xF for comparison: http://xenforo.com/robots.txt
     
    Jake Bunce likes this.
  3. BamaStangGuy

    BamaStangGuy Well-Known Member

    Hmm... how is the redirect handled for /posts/? It's obviously a redirect so why is Google indexing that url instead of /threads/?
     
  4. BamaStangGuy

    BamaStangGuy Well-Known Member

  5. RoldanLT

    RoldanLT Well-Known Member

  6. BamaStangGuy

    BamaStangGuy Well-Known Member

    I don't want that to happen. Google Images provides good traffic. xenForo also doesn't have the asterisk so it looks like they are just blocking the root url?
     
  7. RoldanLT

    RoldanLT Well-Known Member

    Maybe you're right.
     
  8. BamaStangGuy

    BamaStangGuy Well-Known Member

    This is quite the flaw is it not for SEO? My Google Webmaster tools sitemap was not showing the /treads/ urls as being indexed because it was actually indexing /posts/ which is a redirect.

    I'm still confused why Google was indexing those urls when it takes you to a page that uses rel="canonical"
     
  9. RoldanLT

    RoldanLT Well-Known Member

    adwade likes this.
  10. Kevin

    Kevin Well-Known Member

    Do you have your attachments viewable to guests or are you using one of the add-ons to allow them access to the images?
     
  11. BamaStangGuy

    BamaStangGuy Well-Known Member

  12. Kevin

    Kevin Well-Known Member

  13. Jeremy

    Jeremy XenForo Moderator Staff Member

  14. Kevin

    Kevin Well-Known Member

    What is the thought on including the 'Allow: /' entry? My first reaction was what this person responded with...

    http://webmasters.stackexchange.com...ay-to-handle-allow-and-disallow-in-robots-txt

     
  15. Kevin

    Kevin Well-Known Member

    Answer is it depends on the bot. Googlebot respects it following the Disallow entries above it but other bots might not and interpret it as allowing everything, essentially ignoring your explicit Disallow entries.
     
  16. BamaStangGuy

    BamaStangGuy Well-Known Member

    This was fixed in 1.1.5 it seems. Thanks!
     
  17. BamaStangGuy

    BamaStangGuy Well-Known Member

    I still see this happening. I don't use a robots.txt to block /posts/ as it shouldn't be necessary with how xenForo works for /posts/ right?

    Screenshot 2013-10-28 21.23.04.png

    This is a forum that went from vBulletin to xenForo 1.2.2. The last I saw fixes to prevent /posts/ from showing up happened in 1.1.5.

    Anyone else have any insight on why Google is still indexing /posts/ with what exists in the xenForo code base?
     
  18. Jeremy

    Jeremy XenForo Moderator Staff Member

    Regardless of Google hitting /posts/ or not, it should be redirecting using a 301 Moved Permanently header, which Google (and other engines) should respect and update their indexes accordingly.

    Have you attempted the robots.txt entry?
     
  19. BamaStangGuy

    BamaStangGuy Well-Known Member

    I used to use it but if xenForo is using a correct 301 and it is still being indexed then something else needs to be figured out here. Google isn't perfect and Matt Cutts is available to talk to here. Something isn't right. These urls are always 301 redirected correct? There is no circumstance where these urls exist where there is not a 301 tagged to them? Therefore, they should not be indexed at all and the correct url, with the canonical tag on it, should be indexed.
     
  20. Jeremy

    Jeremy XenForo Moderator Staff Member

    No, there is no circumstance where a /posts link is not given a proper 301. If Google ignores the redirect and indexes it improperly, there's not much XenForo can do except redirect when it is hit. I also believe any /posts/ link is nofollow'd by default.
     
    BamaStangGuy likes this.

Share This Page