XF 1.1 Define Bot access

abdfahim

Well-known member
My Google crawling errors are increasing everyday :(

One problem I noticed is that Google is giving 403 error (no permission) for some attachments like below.
http://www.talk2know.com/attachments/summary_route-jpg.123/

Now, I check that this page indeed needs login. So, how to handle this issue? Can I define special access group for BOTs (i see this option in my old PHPBB forum)?

Thanks,
 

MagnusB

Well-known member
You shouldn't allow google access to content guest users aren't allowed to see. Rather add Disallow *: /attachments/ in your robots.txt. That way Google wont index your attachments.
 

abdfahim

Well-known member
You shouldn't allow google access to content guest users aren't allowed to see. Rather add Disallow *: /attachments/ in your robots.txt. That way Google wont index your attachments.
you mean I should write following in my robot.txt?
Code:
Disallow *: /attachments/
 
Top