1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 Crawl Errors

Discussion in 'Troubleshooting and Problems' started by snopro, May 21, 2013.

  1. snopro

    snopro New Member

    I've tried to fix some of these with robots.txt but for some reason it doesn't seem to work. Here's my robots file
    http://www.snowboardprogression.com/robots.txt

    I'm still getting these errors...anyone have any ideas?:notworthy:

    403 Errors:
    forum/index.php?search/
    forum/index.php?search/&type=post
    forum/index.php?members/
    forum/index.php?members/membername/
    forum/index.php?threads/threadname/reply
    forum/index.php?attachments/attach.jpg/
    forum/index.php?threads/threadname/reply&quote=26
     
  2. Steve F

    Steve F Well-Known Member

    With the way you have your robots.txt, you may need to turn on Friendly URLs in the ACP

    Options -> Search Engine Optimization -> "Use Full Friendly URLs"
     
    CyclingTribe likes this.
  3. snopro

    snopro New Member

    Thanks I'll give that a try
     
  4. snopro

    snopro New Member

    Still getting some of these same errors. :(
     
  5. MattW

    MattW Well-Known Member

    Have you got access to Google Webmasters? You can specify the URL and have it fetched as Google bot to see exactly what is happening.
     
    Steve F likes this.
  6. snopro

    snopro New Member

    403 errors. I have those pages disabled for guests which is why I wanted to block them via robots.txt but for some reason it's not working and they are still being crawled.
     
  7. Tracy Perry

    Tracy Perry Well-Known Member

    Try putting
    Code:
    Disallow: /index.php?account/
    Disallow: /index.php?find-new/
    Disallow: /index.php?help/
    Disallow: /index.php?goto/
    Disallow: /index.php?login/
    Disallow: /index.php?lost-password/
    Disallow: /index.php?misc/style/
    Disallow: /index.php?online/
    Disallow: /index.php?recent-activity/
    Disallow: /index.php?register/
    Disallow: /index.php?search/
    in your robots.txt, editing to fit the links you are getting.
     
  8. Steve F

    Steve F Well-Known Member

    Whats the date on the errors?

    Your setup should be working, do as Matt suggested, the errors may be old and you may just need to mark as fixed in Google Webmaster Tools.
     
    CyclingTribe likes this.

Share This Page