Is it even possible to stop Facebook bots from scanning my forum? I don't want them near my site yet every time I ban a Facebook IP address or range, they come up with ten new ones. It seems like a never-ending, annoying task. Same with Amazon too.
XF 2.2 How to stop Facebook from scanning forum
- Thread starter Dixie McCall
- Start date
Solution
Use robots.txt. Google has an intro to it if you're not familiar with it. And the Known Bots add-on by @Sim here (free and worth installing) can help identify bots and get the useragent information for adding them to robots.txt.
Known bots page in Resources: https://xenforo.com/community/resources/known-bots.6489/
For Amazon information on adding them to robots.txt is at these links per Known Bots:
Facebook's bot page does not seem to have information on adding it to robots.txt (surprised? I'm not) and I haven't found it yet. @Sim?
Known bots page in Resources: https://xenforo.com/community/resources/known-bots.6489/
For Amazon information on adding them to robots.txt is at these links per Known Bots:
Facebook's bot page does not seem to have information on adding it to robots.txt (surprised? I'm not) and I haven't found it yet. @Sim?
https://ipinfo.io/AS32934 might help. though might be incomplete.
and then there is this: https://developers.facebook.com/docs/sharing/bot/
and then there is this: https://developers.facebook.com/docs/sharing/bot/
Anytime a link to your site is shared on fb it goes out and fetches the content. XF does the same thing as does most any other software. It's harmless.
Thank you, everyone. @Max Taxable You are correct and it's usually about ten at once. 
Yep and it would defeat the purpose of leveraging FB by posting your URL there, by blocking the fetch. It's not a spider or a scraper, FB isn't indexing our content.Thank you, everyone. @Max Taxable You are correct and it's usually about ten at once.
Similar threads
- Solved
