XF 2.2 How to stop Facebook from scanning forum

[Dixie McCall]

Is it even possible to stop Facebook bots from scanning my forum? I don't want them near my site yet every time I ban a Facebook IP address or range, they come up with ten new ones. It seems like a never-ending, annoying task. Same with Amazon too.

 

[Mendalla]

Use robots.txt. Google has an intro to it if you're not familiar with it. And the Known Bots add-on by @Sim here (free and worth installing) can help identify bots and get the useragent information for adding them to robots.txt.

Known bots page in Resources: https://xenforo.com/community/resources/known-bots.6489/

For Amazon information on adding them to robots.txt is at these links per Known Bots:

https://developer.amazon.com/support/amazonbot

Amazon AdBot

adbot.amazon.com

Facebook's bot page does not seem to have information on adding it to robots.txt (surprised? I'm not) and I haven't found it yet. @Sim?

 

[Chromaniac]

https://ipinfo.io/AS32934 might help. though might be incomplete.

and then there is this: https://developers.facebook.com/docs/sharing/bot/

 

[Max Taxable]

Is it even possible to stop Facebook bots from scanning my forum?

Anytime a link to your site is shared on fb it goes out and fetches the content. XF does the same thing as does most any other software. It's harmless.

 

[Dixie McCall]

Thank you, everyone. @Max Taxable You are correct and it's usually about ten at once.

 

[Max Taxable]

Thank you, everyone. @Max Taxable You are correct and it's usually about ten at once.

Yep and it would defeat the purpose of leveraging FB by posting your URL there, by blocking the fetch. It's not a spider or a scraper, FB isn't indexing our content.