Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
I currently have over 100,000 guests at my website, marathonstadium.com
I've made the site require a login for guests, but it hasn't stopped them.
Sorry to ask what's probably been asked a million times, but what's currently the best way to stop or discourage them? I've been searching solutions, but reading this thread it sounds like a lot of them are already outdated.
I've made the site require a login for guests, but it hasn't stopped them.
Sorry to ask what's probably been asked a million times, but what's currently the best way to stop or discourage them? I've been searching solutions, but reading this thread it sounds like a lot of them are already outdated.
Do you use cloudflare? That's the best start to stop the bulk of them. Most of them can't solve cloudflare challenges.
Set up a custom rule that challenges the paths they are going to most, but excludes logged in members and known bots (like google) so members don't get a challenge. And maybe take it from there.
Eg
(starts_with(http.request.uri.path, "/whats-new/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "")
The last bit, referer eq "" means real guests don't get a cloudflare challenge either (they have a "referer" from google or the home page, whereas bots go direct to the path. However some bots fake having a referer so it could let some through - you could try with it first - that bit could always be taken out. The only real guests that would get challenged are those clicking on whats-new.
To stop approximately 80% of the problem, you need to use cloudflare as your DNS, being the simplest of changes to make if you aren't using them already.
Then, you turn on all their bot fighting systems, then add into their WAF security (firewall) to block any countries of zero traffic interest to you (China is prob #1 as they don't really allow outside websites to China but have a lot of abusive stuff coming out of it), then you need to target the things you can right now, headers of fake browsers pretending to be real, etc.
For a while now, I've been building my own software stack, and will soon leave XF and import it into my own stack. That I have far more control over, and much more ease to modify it, so I have much better anti-bot systems within it. It is lightweight and only contains what I need it to contain, no reliance on third party paid systems, and also runs faster on half again of my current server resources, which can also run approx 10x above my current load. I also don't need to wait for updates or deal with further complicated system design catering the masses. My point, it's getting to a place where Gov probably isn't going to do much of anything to fix the issues already happening and can pretty much foresee happening as AI integration becomes worse. We aren't far away from needing AI firewalls to fight AI penetration.
Yes block China as a country. I also blocked Malaysia and Myanmar.
Don't block Singapore as a country as Google and others can come from there - you don't want to block those.
For Singapore, I have these ASN numbers for big hosting/cloud centres blocked and that seemed to stop them from Singapore.
Singapore 139341 (Tencent), 136907, 141180, 151610 (Huawei) 134963 (Alibaba)
Thailand 137876 (Tencent again)
US 45102 (Alibaba cloud - global)
That should stop a lot to start with, but not the residential proxy bots which come from every country.
The custom rule suggested above will help with those as a start.
Don't block Singapore as a country as Google and others can come from there - you don't want to block those.
For Singapore, I have these ASN numbers for big hosting/cloud centres blocked and that seemed to stop them from Singapore.
Singapore 139341 (Tencent), 136907, 141180, 151610 (Huawei) 134963 (Alibaba)
Thailand 137876 (Tencent again)
US 45102 (Alibaba cloud - global)
That should stop a lot to start with, but not the residential proxy bots which come from every country.
The custom rule suggested above will help with those as a start.
