• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.1 Crawl Errors

#1
I've tried to fix some of these with robots.txt but for some reason it doesn't seem to work. Here's my robots file
http://www.snowboardprogression.com/robots.txt

I'm still getting these errors...anyone have any ideas?:notworthy:

403 Errors:
forum/index.php?search/
forum/index.php?search/&type=post
forum/index.php?members/
forum/index.php?members/membername/
forum/index.php?threads/threadname/reply
forum/index.php?attachments/attach.jpg/
forum/index.php?threads/threadname/reply&quote=26
 

Steve F

Well-known member
#2
With the way you have your robots.txt, you may need to turn on Friendly URLs in the ACP

Options -> Search Engine Optimization -> "Use Full Friendly URLs"
 

MattW

Well-known member
#5
Have you got access to Google Webmasters? You can specify the URL and have it fetched as Google bot to see exactly what is happening.
 
#6
Have you got access to Google Webmasters? You can specify the URL and have it fetched as Google bot to see exactly what is happening.
403 errors. I have those pages disabled for guests which is why I wanted to block them via robots.txt but for some reason it's not working and they are still being crawled.
 

Tracy Perry

Well-known member
#7
Still getting some of these same errors. :(
Try putting
Code:
Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/
in your robots.txt, editing to fit the links you are getting.
 

Steve F

Well-known member
#8
403 errors. I have those pages disabled for guests which is why I wanted to block them via robots.txt but for some reason it's not working and they are still being crawled.
Whats the date on the errors?

Your setup should be working, do as Matt suggested, the errors may be old and you may just need to mark as fixed in Google Webmaster Tools.