Zwielicht
New member
Over the last couple of months as my forum has grown, I've had certain bots swarming the site all at once. I have my forum on a VPS that's pretty decent and can handle a moderate amount of traffic, but not 500+ bots visiting at the same time on top of the members and guests.
It got to the point where it was consistently slowing my site down, so I started looking for threads here on the Xenforo forum to see if anyone else has this issue. In the few threads I found, some of the same problematic bots were mentioned, with some of these even ignoring robots.txt. I opted to block them primarily via Cloudflare. If you don't have Cloudflare, you can try blocking them via robots.txt, .htaccess, or see if your web host has a bot management tool like mine does.
I made a list of the bots I blocked and the reasons for it, just in case this helps anyone else out. I personally considered these bots non-essential (i.e., they aren't indexing for search engines or anything that benefits my site).
It got to the point where it was consistently slowing my site down, so I started looking for threads here on the Xenforo forum to see if anyone else has this issue. In the few threads I found, some of the same problematic bots were mentioned, with some of these even ignoring robots.txt. I opted to block them primarily via Cloudflare. If you don't have Cloudflare, you can try blocking them via robots.txt, .htaccess, or see if your web host has a bot management tool like mine does.
I made a list of the bots I blocked and the reasons for it, just in case this helps anyone else out. I personally considered these bots non-essential (i.e., they aren't indexing for search engines or anything that benefits my site).
- Bytespider - This is by far the worst one. It's TikTok's bot and it's just used to train their language learning models. Honestly, just blocking this one alone is worth it.
- Amazonbot - This one is also pretty egregious. Unless you want to allow it because you've found people use Alexa with your site, it's worth blocking.
- Semrush - unless you're using the tool or plan on selling your forum in the future, it's worth blocking this one.
- Ahrefs - same as the above, unless you're using the tool, there's no reason to let them crawl your site as you're not getting anything out of it.
- Mozlinkexplorer - same as the above. This one wasn't as bad as the others, but I blocked it anyway.
