XF 1.4 Google bots, and others ....

[gajinoz]

How do I stop bots like google and others from seeing a particular category and/or forum? I'm happy for Google to index most of my site but not some private areas.

 

[gajinoz]

I think I may have found it. I have the category permissions set to NO for unregistered users. I'm assuming this also includes bots, is this correct?

Also, if I set NO to the category, does this apply to all forums under that or do I need to set each forum individually?

Thanks.

 

[Paul B]

Bots are the same as guests, so yes, they will not have access if you restrict it for guests.

Permissions cascade down, but can be overridden in child nodes by being explicitly set.

 

[Newt]

So if a thread content view is revoked for guests, is a bot following a thread link also redirected to the login/registration page? Or does it get an error message (like a 404)?

Thread content restriction might entice people to register, but does anybody knows how it is handled by a bot, for instance, if seeing titles but not content has some positive or negative effects on SEO (other that not reaching content prevents from indexing it)?
Thanks

 

[Paul B]

The standard error_with_login template would be shown, which is a 403 (forbidden).