1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.2 Googlebot found an extremely high number of URLs on your site

Discussion in 'Troubleshooting and Problems' started by estranged, Aug 14, 2013.

  1. estranged

    estranged Well-Known Member

    I have just received this email from google.

    Most of the urls listed are like

    /find-new/4586453/threads
    /find-new/4649926/threads?page=2

    However find-new is already disallowed in my robots.txt

    Code:
    User-agent: baiduspider
    Disallow: /
    User-agent: *
    Disallow: /community/find-new/
    Disallow: /community/account/
    Disallow: /community/attachments/
    Disallow: /community/goto/
    Disallow: /community/posts/
    Disallow: /community/login/
    Disallow: /community/admin.php
    Allow: /
    Any ideas?
     
    yavuz likes this.
  2. yavuz

    yavuz Well-Known Member

    I keep getting them too. I'm curious what .htaccess code we have to put in to disallow them.
     
  3. MagnusB

    MagnusB Well-Known Member

    Have you checked those URLs in Webmaster tools against the blocked URL tool?
     
  4. tafreehm

    tafreehm Well-Known Member

    I've received similar notification few days back...
     
  5. CreationNation

    CreationNation Active Member

    I just received this as well today on my site (still on another platform and will be moving to XF soon). My site has been around ages and I'm not sure why all of a sudden now I am getting this (I have almost 9 million posts).
     
  6. DRaver

    DRaver Active Member

    Any news with this problem?
     
  7. estranged

    estranged Well-Known Member

    I realized that my robots.txt was wrong. I had copied it from xenforo.com/robots.txt but forgot to remove the "community" folder. I have my forums in root :)

    After fixing the robots.txt and using the correct path I haven't received such alert again from google.
     
  8. tafreehm

    tafreehm Well-Known Member

  9. Brogan

    Brogan XenForo Moderator Staff Member

    Stuart Wright likes this.
  10. Adam Howard

    Adam Howard Well-Known Member

  11. tafreehm

    tafreehm Well-Known Member

    oh yes!! How can I not see that :s. Thanks
     

Share This Page