Crazy amount of guests

puterfixer said:
On that point: what would be the low hanging fruit to gather some sort of daily statistics/analytics about previous day's visitors, grouped by number of requests per ASN? I'd like to be able to check in the morning if there are some new major sources of traffic from some datacenter or VPN and add it to the block rule, without digging through logs manually. I'm not a master of the free CF analytics but they'd have the ability to derive ASNs; the shared hosting has its own limitations for configuration and logging (partial anonymizing of IPs) to avoid the privacy obligations in GDPR, and a XF plugin would need to query some public service to find the ASN of every visitor's IP and still fit within the reasonable limits of a free service.
Click to expand...
The latest Versions of IP Threat Monitor do check the ASN locally via the free Maxmind Lite DB and write it to the log. If you use Proxycheck with it this results in new, unblocked nasty ASNs (and basically only those) popping up in the proxycheck Dashboard. You can filter the logs in ACP pretty well now (including searching for ASNs) and also download them as csv for a deeper analysis. On a not too busy forum filtering the list in ACP and doing a very rough scrolling is enough to show if something uncommon is going on. If you have a solid set of blocked ASNs and countries API requests to proxycheck are relatively low and, on a small forum, possibly below the free 1000 per day mark: One API request per IP that is not handeled locally already (so neither whitelisted not blacklisted by IP, ASN or country, not iCloud Private relay and neither belonging to any of the published search engine crawler ranges nor used by a logged in member). So the number of API calls really depends from the number of unblocked guests and from how long you preserve the automatically built up blacklist and access list. API requests are pretty cheap anyway, so a paid account does not hurt financially.

Combined with the free Bot Guard add on you have a pretty good setup for shared hosting (and not only for shared hosting): Threat Monitor does the rough filtering and reduces the bad traffic massively while Bot Guard does the more clever bits after that and stopps a vast amount of bots that are left from ASNs you don't want to block. Through the traffic reduction by the first filter stage by bot guard this leaves server load painless.

The combination of both add ons is a pretty nice setup, easy to use, runs on shared hosting and w/o cloudflare. The only active work is to build up your filter list of countries and ASNs as well as your whitelist in the IP Threat Monitor configuration. You can howeve combine both tools with CF as well if you want.
 
Last edited:
Gonna say... pretty happy with my current ruleset. My traffic is now normal, 165k uniques, around that 150k normal daily, and WAF events are doing their job, getting rid of 130k of cf.client.bot useless crawlers from my site. That browser impersonator... still the best rule to date, it has 9 solves from 280k events... that is the single most effective rule I have ever employed to my knowledge.

Screenshot 2026-10-08 072030.webp

My current browser impersonator rule:

Code: 
(
  (
    (
      http.user_agent contains "Chrome/1"
      and not any(lower(http.request.headers.names[*])[*] == "sec-ch-ua")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
      and not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
      and not any(lower(http.request.headers.names[*])[*] == "accept-language")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and http.user_agent contains "Version/"
      and http.user_agent contains "Safari/"
      and not (http.user_agent contains "Chrome/")
      and not (http.user_agent contains "Chromium/")
      and (
        not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
        or not any(lower(http.request.headers.names[*])[*] == "accept-language")
      )
    )
  )
  and not (
    http.cookie contains "xf_user="
    or http.cookie contains "xf_session="
  )
)
 
Just my 2 cents for improvement: it should be named something more fitting, like "You have no power here" :LOL:

On the forum I switched entirely private and blocked all bots, I just found out that a bunch of AI bots are desperately trying to find if I've been a stupid vibe coder and published anywhere my file with passwords, secrets, keys etc. Fun times.
 
Finally behind cloudflare with some decent rules in place but these AI bots are still running upwards of 10K guests on a board that rarely has more than 10 registered users on at a time and it's slowing down the board on a regular basis. Short of blocking the site off to the outside entirely I have come to the conclusion that I just have to accept it.
 
TheGroove said:
Finally behind cloudflare with some decent rules in place but these AI bots are still running upwards of 10K guests on a board that rarely has more than 10 registered users on at a time and it's slowing down the board on a regular basis. Short of blocking the site off to the outside entirely I have come to the conclusion that I just have to accept it.
Click to expand...
Possibly your CF rules are not so decent then? Within this thread there are a bunch of examples with and w/o using CF that should be able to bring down the number of bots way below of what you face.
 
TheGroove said:
Finally behind cloudflare with some decent rules in place but these AI bots are still running upwards of 10K guests on a board that rarely has more than 10 registered users on at a time and it's slowing down the board on a regular basis. Short of blocking the site off to the outside entirely I have come to the conclusion that I just have to accept it.
Click to expand...
What rules have you got so far? I have one block rule (countries and asn's and some specific bots), and two challenge rules. One for uri paths and one for certain countries. But I also have botfightmode turned on - you could maybe turn that on if you haven't already, as it catches a lot before they even get to your rules. That got me down from 5,000 to about 250 to 500 on average, most of the time.

I'd recommend country blocking China, Malaysia and Myanmar unless you have users there.

I also turned on AI labyrinth and precursor. And set a rule for precursor.

For Singapore, I found blocking these ASN's stopped Singapore appearing - they are all cloud centres so don't affect regular users - you don't want to block the country IMO as google and other search engines come from there:

Singapore:
Tencent 139341 (Aceville)
Alibaba 134963
Huawei Cloud International 151610

Thailand:
Tencent in Thailand: 137876

If you don't want to block China as a country then these are the ASN's I found that were the biggest culprits. Cloud centres again mainly.

China:
Tencent: 132203, 45090, 133478 (Comsenz), 132591 (Comsenz), 58835 (Tencent cloud)

Alibaba: 37963, 38369 (Alibaba Taobao) 45104 (Alibaba Cloud). 24429 (Alibaba Cloud Global Content Delivery network).

Huawei: 141180, 136907

Also:

China Telecom 140308

China Mobile 132525

CSTNET 7497 (Virulent)

Easier just to block China.

Hope that helps. Obviously helps to look up the ASN's yourself and double check rather than just rely on someone else's info :-)
 
Last edited:
Just wanted to share something in case it helps anyone. I previously had Thailand blocked. As my numbers had been right down for a while and there could be some potential members from Thailand, I decided to move it to a challenge rule. Gradually numbers started creeping up again both hitting the site and getting through. A peak of 35,000 (although only about 1100 in the guest list but up from 150!) Not necessarily all from Thailand itself. According to google, Thailand is currently full of data centres - a big hub. Or it has a lot of residential proxies there.

So just to say, in case blocking Thailand helps anyone else :-) I'm blocking it again - annoying as I don't like blocking countries. There are so many ASN's there that I'm not going to look them all up right now.

Since blocking - and within 5 minutes, the numbers in the guest list immediately halved again and back down to 500. They had been stable at 150 to 250 for some time. Unblocking the Philippines first raised them to 500. Sat with that for a bit then unblocked Thailand. Both will have to go back in the block - although I can live with 500 guests so might leave Philippines unblocked.
 
Last edited:
This one rule has eliminated 1.36 million in scraper traffic on my forum
1791547281152.webp


as far as countries go , Allowing only these countries on your forum will solve atleast 75% of your bot problems from the start.

1791547441227.webp
 
Depends on the audience. According to your last rule, I'm not in those 6 whitelisted countries and, to my knowledge, I'm not a known bot, so I'd get 403'd on your site.
 
puterfixer said:
Depends on the audience. According to your last rule, I'm not in those 6 whitelisted countries and, to my knowledge, I'm not a known bot, so I'd get 403'd on your site.
Click to expand...
That is correct , I get very little value on my niche from international traffic, Those countries listed in my setup are the countries that bring in the best revenue. After 25 years running a website I am just basically in it for the money and the clean traffic now.
 
greenchicken said:
This one rule has eliminated 1.36 million in scraper traffic on my forum
Click to expand...

That's an interesting collection of ASNs, but somehow a bit weird at the same time.

AS7303 #, AR, (Telecom Argentina S.A.)
AS7979 #, RU US NL, Servers.com Inc. Ace Data Centers, Inc.
AS14061 #, US IN INT, Digital Ocean
AS24940 #, De FI, Hetzner
AS30633 #, US, LEASEWEB-USA-WDC
AS48090 #, AND, TECHOFF SRV LIMITED -> small
AS132203 #, SG HK US ID JP int, TENCENT-NET-AP Shenzhen Tencent Computer Systems Company Limited
AS401560 #, US, (OneCable Network LLC) -> DSL/Cable

AS23033 US Wowrack -> https://bgp.he.net/AS23033
AS26863 US Gameserverkings -> https://bgp.he.net/AS26863
AS60630 FR Association Alsace Reseau Neutre -> https://bgp.he.net/AS60630
AS212317 DE Hetzner -> https://bgp.he.net/AS212317

While there are some clear biggies in terms of bots in there some of these ASNs are blocked through your country rules anyway and a lot of biggies are missing. I do have a pretty excessive ASN blacklist (650+ ASNs) but have not blocked the last four in my list as I do not have had any traffic from them. On the other hand you missed out on loads of US based server ASNs that routinely send legions of bots while you block a dialup ASN (AS401560) that may suffer from a lot of resident proxies, but probably no worse than other US based dialup providers (and there are loads of them) and, if you have US audience, you'll probably block genuine audience as well when blocking this ASN completely.

Clearly, whit limiting your audience to only a hand full of white listed countries you pretty effectively block a lot of the bad traffic and if you can live with the collateral damage there is nothing that would speak against it. Sadly, having whitelisted the US, which is the by far biggest source of bad bots, you have a huge hole in your protection. There's a lot of improvement possibilities in the setup to lock out legions of bots from the US w/o hurting your general audience.

Not sure about how the rules at CF are executed but I'd assume that your "skip" rules should probably come first b/c probably currently they would not be executed in case there was some overlap with any of the "block" rules that get executed first.
 
Last edited:
smallwheels said:
That's an interesting collection of ASNs, but somehow a bit weird at the same time.

AS7303 #, AR, (Telecom Argentina S.A.)
AS7979 #, RU US NL, Servers.com Inc. Ace Data Centers, Inc.
AS14061 #, US IN INT, Digital Ocean
AS24940 #, De FI, Hetzner
AS30633 #, US, LEASEWEB-USA-WDC
AS48090 #, AND, TECHOFF SRV LIMITED -> small
AS132203 #, SG HK US ID JP int, TENCENT-NET-AP Shenzhen Tencent Computer Systems Company Limited
AS401560 #, US, (OneCable Network LLC) -> DSL/Cable

AS23033 US Wowrack -> https://bgp.he.net/AS23033
AS26863 US Gameserverkings -> https://bgp.he.net/AS26863
AS60630 FR Association Alsace Reseau Neutre -> https://bgp.he.net/AS60630
AS212317 DE Hetzner -> https://bgp.he.net/AS212317

While there are some clear biggies in terms of bots in there some of these ASNs are blocked through your country rules anyway. I do have a pretty excessive ASN blacklist (650+ ASNs) but have not blocked the last four in my list as I do not have had any traffic from them. On the other hand you missed out on loads of US based server ASNs that routinely send legions of bots while you block a dialup ASN (AS401560) that may suffer from a lot of resident proxies, but probably no worse than other US based dialup providers (and there are loads of them) and, if you have US audience, you'll probably block genuine audience as well when blocking this ASN completely.

Clearly, whit limiting your audience to only a hand full of white listed countries you pretty effectively block a lot of the bad traffic and if you can live with the collateral damage there is nothing that would speak against it. Sadly, having whitelisted the US, which is the by far biggest source of bad bots, you have a huge hole in your protection. There's a lot of improvement possibilities in the setup to lock out legions of bots from the US w/o hurting your general audience.

Not sure about how the rules at CF are executed but I'd assume that your "skip" rules should probably come first b/c probably currently they would not be executed in case there was some overlap with any of the "block" rules that get executed first.
Click to expand...
Yes some block rules overlap as I sometime do not see the country it's coming from at the moment but that's not real important because you have plenty of space for expressions. Skip rules are powerful, it's important to get as much as you don't want on your forum blocked before they get a chance to skip anything. I see you list some of the ones I posted as small but small can mean 200k hits per month on my site and I want them gone. USA does have many bad bots but it's far easier to deal with them from the handful of countries instead of dealing with them from the world. If the world put revenue in my pocket I would do things differently. Major advertisers want the majority of your traffic to come from the US , that's where the money is. It's not a hate thing for me, I don't hate on countries but many international countries bring nothing but spam and scraping to a forum and add no value in return. My niche is for USA traffic as 99% of the content is not even related to any countries basically except USA and Canada. Everybody has to do what they think is right for their forum, This is what's right for my forum and makes it easier to catch new offenders down the road
 
Not sure about how the rules at CF are executed but I'd assume that your "skip" rules should probably come first b/c probably currently they would not be executed in case there was some overlap with any of the "block" rules that get executed first.
Click to expand...
You have to make sure your block rules have nothing in them that will block what you are trying to skip to begin with then make sure the things you are trying to skip does not get caught up in super bot fight mode as well
 
greenchicken said:
Skip rules are powerful, it's important to get as much as you don't want on your forum blocked before they get a chance to skip anything.
Click to expand...
That's why their position in the ruleset matters and why I would recommend to check the rule execution order with CD - normally I'd expect the skip rules on top.

greenchicken said:
It's not a hate thing for me, I don't hate on countries but many international countries bring nothing but spam and scraping to a forum and add no value in return. My niche is for USA traffic as 99% of the content is not even related to any countries basically except USA and Canada. Everybody has to do what they think is right for their forum, This is what's right for my forum and makes it easier to catch new offenders down the road
Click to expand...
There's nothing to say agains optimizing for you audience and it is a good idea in my opinion, to block those you are not interested in completely. I've blocked around 100 countries currently, so we are in the same boat more or less. All I'm saying is that within the US traffic you can do a lot of optimization regarding your rules: There's easily 100+ ASNs from the US worth blocking that offer nothing but bot traffic and blocking won't hurt you but safe server resources and lock the bots out.

On the other hand, if you have genuine US traffic I probably wouldn't block dialup networks where this comes from but rather challenge them. Currently you have blocked one dialup network completely and do nothing regarding all the other ones. There are really loads of dialup ASNs within the US and most are completely poisoned with resident proxies. So it makes no sense to block one completely and leave the others alone - seems much more useful to treat them equally which would mean: unblock AS401560 and rather challenge it along with all the other dialup networks that you currently ignore.

In combination this will bring down your bot traffic massively.
 
I actually found a better way for my small forum. Just a Javascript and cookie verification that the visitor is human (I'm using AndyB's Human Verification add-on). With my .htaccess rules, and block bots add-on I was still getting hundreds of bots. About an hour after installing the human verification challenge, it went to zero bots.
 
You must log in or register to reply here.

Similar threads

O
  • Question Question
XF 2.2 Huge amount of guests
Replies
6
Views
189
infinitysnapz
infinitysnapz
Wildcat Media
Unusually high number of guests (4x to 5x normal)
Replies
8
Views
222
Tumpe
Tumpe
Tyson
Possible to limit amount of threads a guest can view?
Replies
4
Views
1K
Junior
J
Back
Top Bottom