Google tells me Cloudflare alternatives are Fastly, Datadome, Sucuri and Imperva and Bunny.net, amongst others. However none have a free tier. Although Bunny.net is pennies to a couple of pounds a month apparently and has DNS, WAF and DDOS protection. I can't see Cloudflare losing the free tier when it has competitors.
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Here's a scary statistic "Non-human traffic passed human traffic on Cloudflare’s network in May" (article is 2026). "Cloudflare CEO Matthew Prince said on TBPN that agent traffic could reach a thousand times human traffic within five years if current trends continue."
It talks about the costs to Cloudflare but seems to be about charging AI rather than users of the service. They have been quoted as being committed to keeping the free tier.
www.tikr.com
It talks about the costs to Cloudflare but seems to be about charging AI rather than users of the service. They have been quoted as being committed to keeping the free tier.
Cloudflare CEO: Agent Traffic Will Be “a Thousand Times Human Traffic.” Its Pricing Has to Change
Somebody has to pay for all those bots
www.tikr.com
Last edited:
"Probably"
. Back in June XF sharpened their CF configuration. They needed to b/c they drained in bots and the forum became slow:Before that, it seemed that there had not been bot-rules in place, given the immense amount of guests (while this thread here was already ongoing for half a year and many of it's participants had been fighting the topic way earlier).
So you are missing the point: XF recommends Cloudflare. Their own forum is their storefront - it should (at least in my eyes) show a "perfect" implementation. What it shows is Cloudflare rules that regularly and constantly annoy genuine, logged in users while browsing the forums, yet still there seem to be 2.000-4.000 bots in the forum at any given moment.
Simply does not seem a good storefront/showcase to me and neither a good equilibrium between protection and annoyance.
@Anthony Parsons put immense amounts of time and brain into his CF setup. His forum does show ~1.400 guest currently, probably most of them bots. Realistically, whenever a forum does show a member/guest ratio of more than 1:10 or 1:20 one can in most cases be pretty safe that these are bots I'd guess.
He does not focus on elminiating the bots but mainly does care about server load while at the same time trying to warrant a good user experience to visitors. He does reach both I'd say. Yet still, given the effort he puts in his rules it seems astonishing, that still such an amount get's trough.
Most forum owners lack awareness, time or abilities (or all three) to operate CF at that level. So while CF no doubt linders load on the server it seems at the same time very tough to really stop scraping. As many proxy sellers only carge for successful connection attempts the scrapers won't stop as long as at least some attempts go through - failed attemps are free for them in many if not most cases.
This is a fun one to answer... large ISP's, talking 100k+ into the millions of connected internet subscribers, will absolutely not care about pot-shot/fly-by abuse that does not affect their own network. The only time a large ISP will care or the likes thereof, is when there is a legal obligation (law enforcement) or a significant internet-wide exploitation/attack taking place originating from their network. Ingress malicious traffic to a large ISPs network tends to be automatically filtered and dumped out.
Medium to small sized ISPs (talking about WISPs, per-town or minor-region ISPs) tend to be much more responsive to their provided abuse contacts. This is primarily due to the fact that they are much more bandwidth constrained and abuse on their networks might cause their DIA provider(s) (upstream) to also question their internet traffic - especially if it is excessive or illegal content being transferred.
Small ISPs will also be much more willing to contact their subscribers exhibiting the abusive traffic and work with them to resolve the issue. Proxy traffic can affect their allocated CIDR block reputation - which isn't good for the ma & pa based ISPs.
As in? Looking for an AIO solution for CDN+WAF, among other services in regards to a free-freemium-paid tiering system?
If you want to go scorched earth on the bots... Cloudflare first and then Anubis (or some sort of proof of work intermediary) to get the most bot free experience. If one doesn't nab them, the nuclear option likely will.
I'd think that, too. I used to work for a small, regional ISP 30 years ago and that's what we did back then while the big ones were already pretty ignorant at that time. Has not changed to the better I assume, rather the opposite. But to be fair: Completely different times back then, nonthelast regarding the amount of traffic.
For the moment I am pretty successful with IP Threat Manager regarding that. But I've started to play with anubis on a test VPS and am still thinking about CF as an option at least as a prepartion for the case if really a DDOS like bot wave hits me.If you want to go scorched earth on the bots... Cloudflare first and then Anubis (or some sort of proof of work intermediary) to get the most bot free experience. If one doesn't nab them, the nuclear option likely will.
BTW: You mentioned way up this thread that you would be willing to do a write up how to configure anubis with XF. Would still be interesed in that.
I need to get around to doing that writeup. I've been exploring the weights system a bit more in depth, as it makes things a bit more lenient to who, what, and where does and does not get PoW'd.For the moment I am pretty successful with IP Threat Manager regarding that. But I've started to play with anubis on a test VPS and am still thinking about CF as an option at least as a prepartion for the case if really a DDOS like bot wave hits me.
BTW: You mentioned way up this thread that you would be willing to do a write up how to configure anubis with XF. Would still be interesed in that.
The moment the ISP takes an action, it can be interpreted as tacit acceptance of responsibility for customers' equipment, and if they believe to be responsible for proxies then they should also be held accountable for other sort of traffic monitoring... It can get into a slippery slope fast, especially in countries where lawsuits are a favorite national pasttime. So why poke the hornet nest if closing the eyes on the problem is much easier.
Unfortunately.
No recent experience but in way earlier years, when an enduser would sent spam emails en masse or had unknowingly become part of a botnet or spreading malware many ISPs would have a word with him or simply lock down his connection. Either until the issue was resolved or simply cancelling the contract. Both situations are not terribly different from the resident-proxy situation.
Also I would assume that the TOS for a typical private internet contract forbid malicious behavior like hacking, spreading malware and other things in that direction (which would cover resident proxies) and problably also renting out the internet connection for money (which would cover resident proxies as their sellers claim to pay the privat users their connections they are using) or to unknown persons, being fully responsible for whatever may be done from this connection (which again would cover resident proxies).
You only see something IF cloudflare has an issue with your request.
My recent logs showed there were the 3 or 4 real guests got stuck in a loop with a Cloudflare challenge. Not good. Most went through fine. What would cause that? This was both interactive and managed challenge. I assume some privacy settings or older browsers?
One in particular got 64 challenges over 8 minutes, trying to open things on the home page. Apparently they had been to the home page before, passed a challenge, assume the cf cookie had expired and they later re-used the open browser window. the page was full of images so instead of a page challenge, the page was already open and they got a challenge for every image on the page. While that was a one off/unusual - it's not good. Claude suggested excluding "data" in the rule so the images didn't create a challenge. I guess extending the time the cf cookie lasts for as well but some people with ipads leave browser windows open for a day or more.
One in particular got 64 challenges over 8 minutes, trying to open things on the home page. Apparently they had been to the home page before, passed a challenge, assume the cf cookie had expired and they later re-used the open browser window. the page was full of images so instead of a page challenge, the page was already open and they got a challenge for every image on the page. While that was a one off/unusual - it's not good. Claude suggested excluding "data" in the rule so the images didn't create a challenge. I guess extending the time the cf cookie lasts for as well but some people with ipads leave browser windows open for a day or more.
Last edited:
I added this to the challenge rule
and not starts_with(http.request.uri.path, "/data/")
and not ends_with(http.request.uri.path, "/cover-image")
Basically my home page is articles with images. According to Claude, when this guest went back to the browser tab, and the cf clearance cookie had expired, they would just get a page with blank holes for pictures, because their browser remembered the page, but when their browser requested the images, cloudflare issued challenges, which weren't visible because it can't issue a challenge page for an image apparently? So the guest just got a page with no images and no challenge to pass.
The lines above are supposed to prevent this, so the page loads, they can click on something and then get the challenge.
Shouldn't be an issue if the home page is the main forum page though, but perhaps worth knowing if the home page is an articles page, even if it's a rare occasion.
and not starts_with(http.request.uri.path, "/data/")
and not ends_with(http.request.uri.path, "/cover-image")
Basically my home page is articles with images. According to Claude, when this guest went back to the browser tab, and the cf clearance cookie had expired, they would just get a page with blank holes for pictures, because their browser remembered the page, but when their browser requested the images, cloudflare issued challenges, which weren't visible because it can't issue a challenge page for an image apparently? So the guest just got a page with no images and no challenge to pass.
The lines above are supposed to prevent this, so the page loads, they can click on something and then get the challenge.
Shouldn't be an issue if the home page is the main forum page though, but perhaps worth knowing if the home page is an articles page, even if it's a rare occasion.
My suggestion as a start
In Cloudflare:
Custom Rule Block:
China as a country (and possibly Malaysia, Myanmar and Russia) if you don't need them.
These ASN's in Singapore
139341 (Tencent), 136907, 141180, 151610 (Huawei) 134963 (Alibaba)
Also
Thailand 137876 (Tencent again)
US 45102 (Alibaba cloud - global)
Custom Rule - Managed Challenge:
starts_with(http.request.uri.path, "/whats-new/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq ""
Then see how far the numbers are down before deciding on further custom rules. You can also add other pathways to that rule - eg /search/ /threads/ /posts/
Had a bit of an unusual short peak this morning in form of this yellow mountain...
... and got curios what may have caused it. Interesting outcome: It was obviously a scraping probe: A series of single requests from countless countries, one per country, all targeting the same URL. It seems as if a scraper tested what is blocked and what not in preparation of a bigger scraping attempt. Too bad for him that the doors were locked.
Along with that it turned out independently that the constantly annoying "friends" that use AS20473 (the constant company, an ASN well worth blocking) are not limited to simple scraping but seem also to use bot identity theft at scale, pretending to be common, well known bots:
... and got curios what may have caused it. Interesting outcome: It was obviously a scraping probe: A series of single requests from countless countries, one per country, all targeting the same URL. It seems as if a scraper tested what is blocked and what not in preparation of a bigger scraping attempt. Too bad for him that the doors were locked.
Along with that it turned out independently that the constantly annoying "friends" that use AS20473 (the constant company, an ASN well worth blocking) are not limited to simple scraping but seem also to use bot identity theft at scale, pretending to be common, well known bots:
Also, not sure if you're familiar with Cloudflare custom rules or not. If not, an easy way to start with setting them up is use the field boxes as in the example below. Then set it for managed challenge.
So choose the pathway (bots often go straight to whats-new to scrape). So the rule is - set a challenge if - bots go to whats-new AND if they are NOT a registered member (xf_user cookie), AND they are NOT a known bot (ie google etc). In my example I also added AND Referer equals (leave blank) (ie any referer). So if a real guest has come from google or has been to the home page before clicking whats-new, they carry a referer from that page, so they bypass the challenge as well, so it's only bots that get challenged. However, some bots (not all) also carry a fake referer so that lets some of them slip through. So it's a personal choice. I did it on that pathway to avoid challenging as many real people as possible. But you can add it in or take it out at any time and see what difference it makes.
Then if you add another pathway to challenge - eg search. It is OR. then repeat - choose pathway, AND not xf_user, AND not known bots
then OR
Third pathway. All of this pre-populates the expression box at the bottom.
Apologies if you already knew that.
Then down the page choose the rule order (after any block rule - block rules need to come first), and choose Managed Challenge, then save/deploy.
And watch the bots get trapped in the graph and statistics next to the rule.
This is all on the free plan. You just sign up for an account. It means switching your DNS to cloudflare if it's not already.
These bot numbers are quite low on my graphs now as many just gave up after being blocked. Ignore all my block rules though - you only need one block rule for ASN's and countries - I just split them up for various reasons. So it tends to cut the numbers right down as soon as you block China and those ASN's. The challenge rules are then to catch the ones coming from all the other countries - the residential proxies.
Edit - those solvers are being caught by my honeypot and not getting through. It's rare that a real guest gets to the honeypot, but if they do they just click a button to continue. The reason the honeypot block shows 0 is because the bots can't click on the button or solve the honeypot so just get stopped there. If I didn't have the honeypot I'd probably have about 1000 requests regularly on the site as one big swarm solves challenges, and if a big swarm came back again that could easily get up to 2 or 3,000 again. 2 or 3,000 is manageable - I just would prefer to keep as many off as possible so they're not scraping my site.
Last edited:
In the first place you cannot be sure if he is using or plans/wants to use Cloudflare at all...
