1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 Define Bot access

Discussion in 'Troubleshooting and Problems' started by abdfahim, Aug 4, 2012.

  1. abdfahim

    abdfahim Active Member

    My Google crawling errors are increasing everyday :(

    One problem I noticed is that Google is giving 403 error (no permission) for some attachments like below.
    http://www.talk2know.com/attachments/summary_route-jpg.123/

    Now, I check that this page indeed needs login. So, how to handle this issue? Can I define special access group for BOTs (i see this option in my old PHPBB forum)?

    Thanks,
     
  2. MagnusB

    MagnusB Well-Known Member

    You shouldn't allow google access to content guest users aren't allowed to see. Rather add Disallow *: /attachments/ in your robots.txt. That way Google wont index your attachments.
     
    abdfahim likes this.
  3. abdfahim

    abdfahim Active Member

    you mean I should write following in my robot.txt?
    Code:
    Disallow *: /attachments/
    
     
  4. MagnusB

    MagnusB Well-Known Member

    yes, if you use full friendly URLs, all attachments will be "stored" there.
     
    abdfahim likes this.

Share This Page