XF 1.2 Xenforo SEO concern

[eagle eyes]

Today i got this in webmasters tool. What can be done to avoid this?

Googlebot encountered problems while crawling your site http://domain.com/.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

More information about this issue

Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.

http://domain.com/find-new/2438494/posts?page=6
http://domain.com/goto/post?id=2394656
http://domain.com/goto/post?id=1592476
http://domain.com/find-new/2696989/posts?page=14
http://domain.com/find-new/3048337/posts?page=4
http://domain.com/threads/274968/page-5
http://domain.com/find-new/2246456/posts?page=6
http://domain.com/threads/another-reality-of-karachi-operation.291105/
http://domain.com/find-new/957749/posts?page=7
http://domain.com/find-new/2572043/posts?page=2
http://domain.com/find-new/2589658/posts?page=6
http://domain.com/find-new/3351862/posts?page=4
http://domain.com/posts/5283894/
http://domain.com/find-new/2604400/posts?page=9
http://domain.com/find-new/3243170/posts?page=5
http://domain.com/find-new/3681677/posts?page=6
http://domain.com/find-new/4099852/posts?page=14
http://domain.com/find-new/4020780/posts?page=2
http://domain.com/find-new/3060238/posts?page=14
http://domain.com/goto/post?id=900443
http://domain.com/find-new/3025463/posts?page=5
http://domain.com/find-new/1475830/posts?page=8
http://domain.com/find-new/1907689/posts?page=2
http://domain.com/goto/post?id=117698
http://domain.com/find-new/3568942/posts?page=2
http://domain.com/find-new/3662413/posts?page=7
http://domain.com/posts/5700685/
http://domain.com/find-new/3157293/posts?page=2
http://domain.com/gallery/external.php?type=RSS2&cat=500&ppuser=26722
http://domain.com/find-new/3361660/posts?page=5
http://domain.com/find-new/3600065/posts?page=5
http://domain.com/find-new/3544835/posts?page=4
http://domain.com/find-new/4130366/posts?page=4
http://domain.com/goto/post?id=2710000
http://domain.com/find-new/4002542/posts?page=6
http://domain.com/find-new/1198023/posts?page=6
http://domain.com/threads/america-bashing-pakistan-to-please-india.183431/page-3
http://domain.com/find-new/2421945/posts?page=5
http://domain.com/goto/post?id=2631815
http://domain.com/find-new/4047107/posts?page=14
http://domain.com/find-new/1467141/posts?page=6
http://domain.com/find-new/3065828/posts?page=5
http://domain.com/find-new/2741523/posts
http://domain.com/find-new/3963118/posts?page=13
http://domain.com/find-new/4038305/posts?page=4
http://domain.com/find-new/2864700/posts?page=4
http://domain.com/threads/205839/
http://domain.com/find-new/1436119/posts?page=7
http://domain.com/find-new/1582678/posts?page=9

 

[Chris D]

You need to add a robots.txt file to instruct search engines not to crawl those links. Just like this one:

http://xenforo.com/robots.txt