XF 1.2 Googlebot found an extremely high number of URLs on your site


Well-known member
I have just received this email from google.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

More information about this issue

Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.
Most of the urls listed are like


However find-new is already disallowed in my robots.txt

User-agent: baiduspider
Disallow: /
User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
Any ideas?


Well-known member
I keep getting them too. I'm curious what .htaccess code we have to put in to disallow them.


Active member
I just received this as well today on my site (still on another platform and will be moving to XF soon). My site has been around ages and I'm not sure why all of a sudden now I am getting this (I have almost 9 million posts).


Well-known member
I realized that my robots.txt was wrong. I had copied it from xenforo.com/robots.txt but forgot to remove the "community" folder. I have my forums in root :)

After fixing the robots.txt and using the correct path I haven't received such alert again from google.