Fixing Google Webmaster problems

[Trix]

Hopefully this is the correct place to post this, as it's non xenforo related.

I have my OLD ipb site at /forum/ and my new xenforo site at /forums/ with a redirect provided by Xenforo. So if you visit /forum/ you get sent to /forums/

Ok. In google webmaster on the left, if i go to Health - Crawl errors i have a list down the page of URL errors. 2,980 to be precise. They all seem to be errors from my old IPB site. I'm not sure if this is normal, or not, or if it even matters really.

They are showing up as 503 errors. Links such as

/forum/index.php/user/32323-username

Not knowing what i am really doing, to me i'm worried this is somehow effecting my search standings, or something like that with google. I need to keep my old forums alive for a while so that i can retrieve photos from the old gallery if people request them. But really, i don't want google seeing it at all, kind of invisible. Is this possible? I thought about maybe adding that directory to the robots text, but then thought that might hurt the redirect script somehow, or any results in google that have not updated yet.

Hope all this makes sense and someone with more knowledge than me can help. Thanks everyone!

 

[craigiri]

I think the key is how the number of errors relates to the totality of your redirects and site traffic....

That is, I have about 150 errors per day - mostly from funny URL's left over from my old forum, but that is against 80,000 page views (and, obviously, more total file calls) on the same day.

Personally, I don't worry about this tiny proportion......it has not affected any search, as my organic traffic is actually up since my move to XF.

 

[DRE]

I regret getting rid of my vB4 installation. I wish I saved it.

Save your old IPB installation. You may need it later.

I don't mean save as in so that you can move back to it, but to retrieve data that might come in handy.

 

[TheBigK]

I've over 90k of them and I've fixed most of them already; but Google Webmaster Tools takes it's own sweet time to acknowledge that the errors have been fixed. It's currently dropping them from its report at the rate of 1000 per day. So maybe I've a long time to go before Google discovers the errors are all gone.

Here's a tip: As soon as you fix an error - be sure to mark it as 'fixed' in GWT.

 

[TheBigK]

Important Update:

Recently Google Webmaster Tools reported the following -

Googlebot encountered problems while crawling your site http://www.crazyengineers.com/.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

More information about this issue

Here's a list of sample URLs with potential problems. However, this list may not include all problematic URLs on your site.

http://www.crazyengineers.com/community/forums/ce-infocus.52/page-204?order=view_count
http://www.crazyengineers.com/community/find-new/3047035/threads
http://www.crazyengineers.com/community/find-new/1251351/threads?page=6
http://www.crazyengineers.com/community/find-new/2601881/threads?page=3
http://www.crazyengineers.com/community/find-new/3490322/threads?page=6

Now I downloaded all the errors from GWT and found out that Google's found over 100,000 of such URLs on the site -

http://www.crazyengineers.com/community/goto/post?id=215874
http://www.crazyengineers.com/community/goto/post?id=133680
http://www.crazyengineers.com/community/goto/post?id=220205
http://www.crazyengineers.com/community/find-new/3490322/threads?page=6
http://www.crazyengineers.com/community/find-new/3733339/threads?page=2
http://www.crazyengineers.com/community/find-new/671850/threads?page=10
http://www.crazyengineers.com/community/posts/56658/
http://www.crazyengineers.com/community/posts/56859/
http://www.crazyengineers.com/community/posts/56657/
http://www.crazyengineers.com/community/posts/55927/
http://www.crazyengineers.com/community/posts/55602/
http://www.crazyengineers.com/community/posts/55925/
http://www.crazyengineers.com/community/posts/56858/
http://www.crazyengineers.com/community/posts/56860/
http://www.crazyengineers.com/community/posts/56660/
http://www.crazyengineers.com/community/posts/56870/
http://www.crazyengineers.com/community/posts/55924/
http://www.crazyengineers.com/community/posts/55928/
http://www.crazyengineers.com/community/posts/233962/
http://www.crazyengineers.com/community/posts/236973/
http://www.crazyengineers.com/community/posts/236671/
http://www.crazyengineers.com/community/posts/235698/

I read Google's recommendation on solving this problem: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=76401

As suggested by members, I had updated my robots.txt to stop Google bot from following /community/posts/ URLs, but it looks like Google's already indexed them!

Can someone tell me what should I do next? Just wait and watch? Also, after taking the corrective steps, I notice that the site isn't producing more errors and dropping them at the rate of 1000/day as I mark them 'fixed' in Google Webmaster Tools.

What should be my next step?

 

[MagnusB]

As suggested by members, I had updated my robots.txt to stop Google bot from following /community/posts/ URLs, but it looks like Google's already indexed them!

Can someone tell me what should I do next? Just wait and watch? Also, after taking the corrective steps, I notice that the site isn't producing more errors and dropping them at the rate of 1000/day as I mark them 'fixed' in Google Webmaster Tools.

What should be my next step?

You can ask for removal of the posts/ folder, but according to Google it should only be used where it is "urgent" to remove them (e.g. sensitive information is visible on the page). I think it would be prudent and wait until Google sorts this out by them self, and eventually update you through GWT.

 

[TheBigK]

You can ask for removal of the posts/ folder, but according to Google it should only be used where it is "urgent" to remove them (e.g. sensitive information is visible on the page). I think it would be prudent and wait until Google sorts this out by them self, and eventually update you through GWT.

Thanks, MagnusB. I just observed our index status and it looks like Google's dropped them from Index after robots.txt blocked them. I'm wondering if this is the reason we were dropped in traffic.

 

[MagnusB]

Google moves in mysterious ways