Googlebot found an extremely high number of URLs on your site

[TheBigK]

Looks like our affair with the Google Bot isn't ending. I checked Google Webmaster Tools and it's reporting -

Googlebot found an extremely high number of URLs on your site
Googlebot encountered problems while crawling your site http://www.crazyengineers.com
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.


Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.

• http://www.crazyengineers.com/community/forums/ce-infocus.52/page-204?order=view_count
• http://www.crazyengineers.com/community/find-new/3047035/threads
• http://www.crazyengineers.com/community/find-new/1251351/threads?page=6
• http://www.crazyengineers.com/community/find-new/2601881/threads?page=3
• http://www.crazyengineers.com/community/find-new/3490322/threads?page=6
• http://www.crazyengineers.com/community/find-new/3733339/threads?page=2

I've absolutely no clue what's going wrong and in the last ~11 months of running Xenforo, I've never had this issue. I didn't make any change to the website that could have resulted in this issue.

Can someone inspect our site and see if anything need to be fixed?

 

[TheBigK]

Well it looks like the other shoe dropped because I just lost 60-70% of my Google traffic. However my crawl stats are somewhat back to normal now.

Did you do anything to get your crawl stats back? I've adjusted the setting in GWT but it still is at what it was. I want Google to crawl my web-site faster and discover that all the URLs have been fixed.

 

[melbo]

When moving from /forum/ to root, you should always

I started with 120K++ 404 errors before 10 months and I am still on 11K 404 on one of my website as I changed my forums from /forum/ directory to root directory.

After one of the Google updates my e-commerce website got hit and I am trying to fix the issues. I noticed resolving duplicate title content helps.

The one thing I would suggest is to block "Baidu" bots and increase Google crawl frequency which would compensate usage on your VPS or Dedi server. That worked well for my friend which I mentioned in another topic of yours. He recovered within 3 months and I think he is back to stable as per organic traffic is concerned.

When moving from /forum to root, you should have created 301 (permanent) redirects and Google wouldn't have skipped a beat.

 

[hellreturn]

When moving from /forum/ to root, you should always
When moving from /forum to root, you should have created 301 (permanent) redirects and Google wouldn't have skipped a beat.

I believe it was due to IPB FURL structure it was not possible and hence issues with forum sections. Topics and posts where all redirected with 301.

 

[DBA]

Did you do anything to get your crawl stats back? I've adjusted the setting in GWT but it still is at what it was. I want Google to crawl my web-site faster and discover that all the URLs have been fixed.

Nope it came back by itself. Traffic is still down though.

 

[TheBigK]

Nope it came back by itself. Traffic is still down though.

Sorry to hear that. Keep me posted on your progress please. I've read that Google's penalising complete domains if they discover thin content on your website. Our members have created several threads that are practically useless; but are great fun to read and participate in. Not sure what should I do with such content.

 

[DBA]

Sorry to hear that. Keep me posted on your progress please. I've read that Google's penalising complete domains if they discover thin content on your website. Our members have created several threads that are practically useless; but are great fun to read and participate in. Not sure what should I do with such content.

I think I got hit with the domain name keyword thing. My domain/brand name has my primary keyword in it.