XF 1.3 Allow only specifc bots?

[Neil E.]

Based on experiences with other forum software I'm wondering what to do about bots. Other software has a bot list already incorporated. XF does not appear to have this. I believe bots should be treated differently than guests. Should specific desired bots have a bot usergroup with permissions? How do you disallow all other bots? What should be in robots.txt?

Registration will be handled by manual approval with Q&A Captcha and StopForumSpam. This should limit spambot registration.

 

[Paul B]

There are some bots defined in code.

I believe there is an add-on which adds additional ones.

 

[Mike]

I believe bots should be treated differently than guests.

When it comes to Google, giving them access but not real guests runs the risk of being considered "cloaking".

 

[AndyB]

How do you disallow all other bots?

Here you go:

https://xenforo.com/community/resources/robots.3900/

 

[Neil E.]

When it comes to Google, giving them access but not real guests runs the risk of being considered "cloaking".

Argh, so much to learn!

Here you go:https://xenforo.com/community/resources/robots.3900/

Andy, fantastic job!