XF 1.1 Crawl Errors

[snopro]

I've tried to fix some of these with robots.txt but for some reason it doesn't seem to work. Here's my robots file
http://www.snowboardprogression.com/robots.txt

I'm still getting these errors...anyone have any ideas?

403 Errors:
forum/index.php?search/
forum/index.php?search/&type=post
forum/index.php?members/
forum/index.php?members/membername/
forum/index.php?threads/threadname/reply
forum/index.php?attachments/attach.jpg/
forum/index.php?threads/threadname/reply&quote=26

 

[Steve F]

With the way you have your robots.txt, you may need to turn on Friendly URLs in the ACP

Options -> Search Engine Optimization -> "Use Full Friendly URLs"

 

[snopro]

With the way you have your robots.txt, you may need to turn on Friendly URLs in the ACP

Options -> Search Engine Optimization -> "Use Full Friendly URLs"

Thanks I'll give that a try

 

[snopro]

Still getting some of these same errors.

 

[MattW]

Have you got access to Google Webmasters? You can specify the URL and have it fetched as Google bot to see exactly what is happening.

 

[snopro]

Have you got access to Google Webmasters? You can specify the URL and have it fetched as Google bot to see exactly what is happening.

403 errors. I have those pages disabled for guests which is why I wanted to block them via robots.txt but for some reason it's not working and they are still being crawled.

 

[TPerry]

Still getting some of these same errors.

Try putting

Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/

in your robots.txt, editing to fit the links you are getting.

 

[Steve F]

403 errors. I have those pages disabled for guests which is why I wanted to block them via robots.txt but for some reason it's not working and they are still being crawled.

Whats the date on the errors?

Your setup should be working, do as Matt suggested, the errors may be old and you may just need to mark as fixed in Google Webmaster Tools.