Just Some Quick Tips On Blocking Bots

Zwielicht

Zwielicht

Active member
Licensed customer
Over the last couple of months as my forum has grown, I've had certain bots swarming the site all at once. I have my forum on a VPS that's pretty decent and can handle a moderate amount of traffic, but not 500+ bots visiting at the same time on top of the members and guests.

It got to the point where it was consistently slowing my site down, so I started looking for threads here on the Xenforo forum to see if anyone else has this issue. In the few threads I found, some of the same problematic bots were mentioned, with some of these even ignoring robots.txt. I opted to block them primarily via Cloudflare. If you don't have Cloudflare, you can try blocking them via robots.txt, .htaccess, or see if your web host has a bot management tool like mine does.

I made a list of the bots I blocked and the reasons for it, just in case this helps anyone else out. I personally considered these bots non-essential (i.e., they aren't indexing for search engines or anything that benefits my site).
  • Bytespider - This is by far the worst one. It's TikTok's bot and it's just used to train their language learning models. Honestly, just blocking this one alone is worth it.
  • Amazonbot - This one is also pretty egregious. Unless you want to allow it because you've found people use Alexa with your site, it's worth blocking.
  • Semrush - unless you're using the tool or plan on selling your forum in the future, it's worth blocking this one.
  • Ahrefs - same as the above, unless you're using the tool, there's no reason to let them crawl your site as you're not getting anything out of it.
  • Mozlinkexplorer - same as the above. This one wasn't as bad as the others, but I blocked it anyway.
 
I have some info to add to this but it's too late to screen print and stuff I'm about to sleep. Please gimme a nudge mid-week if I've not replied. I'm needing to block whole continents and manage challenge all other countries but my top 2 visitor base. And am stuff having issues.

WTF eh.
 
Donny said:
I have some info to add to this but it's too late to screen print and stuff I'm about to sleep. Please gimme a nudge mid-week if I've not replied. I'm needing to block whole continents and manage challenge all other countries but my top 2 visitor base. And am stuff having issues.

WTF eh.
Click to expand...
Check out this thread - I think you are not alone. I have a country block for 17 countries, multiple ASN blocks and a couple of rules to challenge anything hitting whats new, search or members plus another rule challenging various countries. I wouldn't block whole continents (I was tempted) as you could block good things as well. Challenge them at the site level - the bots these days seem to be using residential proxies from all over the world - if you block a continent they will just come out from a different country instead. Also if you've blocked Asia then best to add to allow known bots so Google can still index you.

Levina

Thread 'Crazy amount of guests'

I have a small forum, usually a few hundred guests at most. The last days the number of guests have been unusually high, around 3000 and climbing. We now have over 4800 guests. That's not normal. Many "Viewing unknown page" or "Latest content", with a warning sign. IP addresses are from all corners of the world. Many from south America, Brazil mostly. Also saw quite a number from Vietnam of all places, but everywhere else too.

We're on Cloud and I don't know what to do. Could we be under attack? We have been in the past, a few times I think. For reasons I cannot possibly fathom; we're a...
  • Like
 
You must log in or register to reply here.
Back
Top Bottom