XF 1.1 Define Bot access

[abdfahim]

My Google crawling errors are increasing everyday

One problem I noticed is that Google is giving 403 error (no permission) for some attachments like below.
http://www.talk2know.com/attachments/summary_route-jpg.123/

Now, I check that this page indeed needs login. So, how to handle this issue? Can I define special access group for BOTs (i see this option in my old PHPBB forum)?

Thanks,

 

[MagnusB]

You shouldn't allow google access to content guest users aren't allowed to see. Rather add Disallow *: /attachments/ in your robots.txt. That way Google wont index your attachments.

 

[abdfahim]

You shouldn't allow google access to content guest users aren't allowed to see. Rather add Disallow *: /attachments/ in your robots.txt. That way Google wont index your attachments.

you mean I should write following in my robot.txt?

Disallow *: /attachments/

 

[MagnusB]

yes, if you use full friendly URLs, all attachments will be "stored" there.