• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.2 Googlebot found an extremely high number of URLs on your site

estranged

Well-known member
#1
I have just received this email from google.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

More information about this issue

Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.
Most of the urls listed are like

/find-new/4586453/threads
/find-new/4649926/threads?page=2

However find-new is already disallowed in my robots.txt

Code:
User-agent: baiduspider
Disallow: /
User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
Any ideas?
 
#5
I just received this as well today on my site (still on another platform and will be moving to XF soon). My site has been around ages and I'm not sure why all of a sudden now I am getting this (I have almost 9 million posts).
 

estranged

Well-known member
#7
I realized that my robots.txt was wrong. I had copied it from xenforo.com/robots.txt but forgot to remove the "community" folder. I have my forums in root :)

After fixing the robots.txt and using the correct path I haven't received such alert again from google.