XF 1.2 Googlebot found an extremely high number of URLs on your site

[estranged]

I have just received this email from google.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

More information about this issue

Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.

Most of the urls listed are like

/find-new/4586453/threads
/find-new/4649926/threads?page=2

However find-new is already disallowed in my robots.txt

User-agent: baiduspider
Disallow: /
User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

Any ideas?

 

[yavuz]

I keep getting them too. I'm curious what .htaccess code we have to put in to disallow them.

 

[MagnusB]

Have you checked those URLs in Webmaster tools against the blocked URL tool?

 

[tafreehm]

I've received similar notification few days back...

 

[CreationNation]

I just received this as well today on my site (still on another platform and will be moving to XF soon). My site has been around ages and I'm not sure why all of a sudden now I am getting this (I have almost 9 million posts).

 

[DRaver]

Any news with this problem?

 

[estranged]

I realized that my robots.txt was wrong. I had copied it from xenforo.com/robots.txt but forgot to remove the "community" folder. I have my forums in root

After fixing the robots.txt and using the correct path I haven't received such alert again from google.

 

[tafreehm]

why following does not work ?
http://xenforo.com/community/robots.txt

@Brogan

 

[Paul B]

It's in the root, where it should be: http://xenforo.com/robots.txt

 

[Adam Howard]

why following does not work ?
http://xenforo.com/community/robots.txt

@Brogan

http://xenforo.com/robots.txt

edit: got beaten to it

 

[tafreehm]

oh yes!! How can I not see that :s. Thanks