I actually just switched my main forum back to centminmod server, just the forum on it, all the business sites I manage are now in CF workers/pages, on the edge, a slow process of getting them away from another piece of hardware. It seems with a new IP they all came looking. Funnily enough, I also added my first CF rule back to see how it would go. Good timing.

You can see I flicked that on last night before quitting for the night after setting up the new server and migration, tuning, etc. What I implemented was this one from my heuristics software, which is the real problem child as I see it, bots faking to be humans:
The solved so far, 258 from 172.73k. Those 258 got no CF managed challenge, nothing, they just passed right through. That is a fairly successful rule I would say, and not even a full 24hrs of it. I got hit by a bit over 400k last night, and that one rule took out 40% of it.
I can see addons have a place for people who don't manage their own VPS, but for everyone who does have root access to their server, addons are the to late option IMO.
You don't need to chase IP's or ASN's, even countries, its just not needed.
I still have bot fight mode off, AI Labyrinth off, all bots not trying to fake being a person, are allowed access still. Obviously if you want to get rid of them, use CF and turn those settings on, along with AI bot blocking, then my heuristics show me a rule applicable to my site that you could add if you want to take out the majority of the nasties, as above, non-interactive challenge.
I still have some tuning to do, but with a 400k+ hit overnight, my server didn't blink. I tuned it roughly at setup from past experience, but I have not fine tuned it yet as I don't have the data from it until a few days. That last peak was me testing my backup script, sending them to S3 and then deleting them off the server, then it was all normal. That 400k hit during those small CPU peaks, barely got the server excited.
IMHO, tuning and DNS firewall is the best solution to this problem, albeit for those who can't do either for whatever reasons. Even downgraded specs, 4 cores 12G ram now, from 6 and 16. Once I fine tune from 48+hrs of data, I will throw some traffic at it to see how it goes.


You can see I flicked that on last night before quitting for the night after setting up the new server and migration, tuning, etc. What I implemented was this one from my heuristics software, which is the real problem child as I see it, bots faking to be humans:
Rule 3 · Challenge the rotating-proxy scraper networks
Your biggest problem: scraper networks using millions of home IP addresses, 1–2 requests each, disguised as normal browsers. Their weakness: they claim OUTDATED browser versions. Your real visitors run current browsers (Chrome 147–149 right now); these networks claim versions months or years old. This rule gives an invisible browser check to outdated-version visitors arriving with no referrer. Bots fail it; a real person on an old browser passes without noticing. Logged-in members are exempt entirely.- 899 stealth campaigns using 8,906,015 IPs made 12,804,480 requests in your data.
- Versions this rule covers (all heavily used by the networks, barely used by your real visitors): Chrome 146, Chrome 133, Chrome 145, Firefox 135, Firefox 133, Chrome 131, Chrome 116, Chrome 124, Chrome 107, Chrome 104, Chrome 110, Chrome 99…
- Traffic covered: about 10,357,845 requests.
Code:
(http.user_agent contains "Chrome/146." or http.user_agent contains "Chrome/133." or http.user_agent contains "Chrome/145." or http.user_agent contains "Firefox/135." or http.user_agent contains "Firefox/133." or http.user_agent contains "Chrome/131." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/99." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/136." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/65." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/117." or http.user_agent contains "Firefox/115." or http.user_agent contains "Chrome/135.") and http.referer eq "" and (not cf.client.bot) and (not http.cookie contains "xf_user")
The solved so far, 258 from 172.73k. Those 258 got no CF managed challenge, nothing, they just passed right through. That is a fairly successful rule I would say, and not even a full 24hrs of it. I got hit by a bit over 400k last night, and that one rule took out 40% of it.
I can see addons have a place for people who don't manage their own VPS, but for everyone who does have root access to their server, addons are the to late option IMO.
You don't need to chase IP's or ASN's, even countries, its just not needed.
I still have bot fight mode off, AI Labyrinth off, all bots not trying to fake being a person, are allowed access still. Obviously if you want to get rid of them, use CF and turn those settings on, along with AI bot blocking, then my heuristics show me a rule applicable to my site that you could add if you want to take out the majority of the nasties, as above, non-interactive challenge.
Code:
(http.user_agent contains "ClaudeBot" or http.user_agent contains "Amazonbot" or http.user_agent contains "meta-externalagent" or http.user_agent contains "SemrushBot" or http.user_agent contains "Bytespider" or http.user_agent contains "GPTBot" or http.user_agent contains "AhrefsBot" or http.user_agent contains "DotBot" or http.user_agent contains "ShapBot" or http.user_agent contains "MJ12bot" or http.user_agent contains "AcademicBot" or http.user_agent contains "meta-webindexer" or http.user_agent contains "OAI-SearchBot" or http.user_agent contains "Claude-SearchBot" or http.user_agent contains "yacybot" or http.user_agent contains "PerplexityBot" or http.user_agent contains "compatible; crawler)" or http.user_agent contains "LinkupBot" or http.user_agent contains "Scrapy" or http.user_agent contains "curl" or http.user_agent contains "Turnitin" or http.user_agent contains "ClueWeb-Crawler" or http.user_agent contains "PurityBot" or http.user_agent contains "ImageBot" or http.user_agent contains "Go-http-client" or http.user_agent contains "YisouSpider" or http.user_agent contains "quillbot" or http.user_agent contains "AwarioBot" or http.user_agent contains "IbouBot" or http.user_agent contains "serpstatbot" or http.user_agent contains "CCBot")
I still have some tuning to do, but with a 400k+ hit overnight, my server didn't blink. I tuned it roughly at setup from past experience, but I have not fine tuned it yet as I don't have the data from it until a few days. That last peak was me testing my backup script, sending them to S3 and then deleting them off the server, then it was all normal. That 400k hit during those small CPU peaks, barely got the server excited.
IMHO, tuning and DNS firewall is the best solution to this problem, albeit for those who can't do either for whatever reasons. Even downgraded specs, 4 cores 12G ram now, from 6 and 16. Once I fine tune from 48+hrs of data, I will throw some traffic at it to see how it goes.


