just got this in my inbox from CF: https://blog.cloudflare.com/monetization-gateway/
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
and here we go:
Cloudflare does not only want to be at the core of the infrastructure, basically deciding who can see what - they now also want to use their position as the middlemen to foster commercialization of content and become the central provider, controlling instance and financial institution that enables and controls all this:
They cover it up in nice words but basically they say: From now on, whatever you do on the net, will cost money. This is what we are working on. Obviously, at the moment they only talk about bots and agents - but it is clear where this is going. And they want to be the company that controls it all and profits from every single transaction. Plus they are probably the only company that, given their current position, is able to do and even enforce something like that. When things go bad, this could become a dystopian monopoly against which Google, Microsoft and Meta look like friendly teddy bears.
Last edited:
Cloudflare definitely will have a stronger hold especially now that server ram and ssd/nvme storage prices are higher - building your own at scale independent of Cloudflare will still be more costly. Especially if predicted memory prices are true - 2x prices by end of 2026, 4.4x prices by 2028 and if 20%/yr increase on 2029/2030, will be at 6x prices by 2030.
I think it's a positive motion, i just don't know when Cloudflare starts thinking+acting like Microsoft or Adobe.
That's why i've optimized my system down to use 225 bytes average per IP before even starting building it, since this is the hardest part.
10 million IP address states would cost you 2.25gb of memory... and we can chuck IP states out every few days.
Any more RAM optimization comes at the price of added CPU consumption per hit, which we have a budget of 0.5ms for.
Once we implement a way to refer a ban upwards to a firewall, that budget can be relaxed and we can reduce that bytes per IP figure.
The database engine, memcached, also has LRU, so you could set a maximum memory size, and it will throw out cold data. That would help overflow. Also it now has extstore so that you can use disk for storage. So theoretically you could store the entire 400m active scraper set if it came down to it. How fast that database performs on-disk is unknown though.
DragonflyDB has lower per-record memory consumption vs memcached. It may be possible to get closer to 175 bytes average per IP address. It can communicate over the memcached protocol. I haven't experimented with it yet.
Lots of room to make it feasible!
That's why i've optimized my system down to use 225 bytes average per IP before even starting building it, since this is the hardest part.
10 million IP address states would cost you 2.25gb of memory... and we can chuck IP states out every few days.
Any more RAM optimization comes at the price of added CPU consumption per hit, which we have a budget of 0.5ms for.
Once we implement a way to refer a ban upwards to a firewall, that budget can be relaxed and we can reduce that bytes per IP figure.
The database engine, memcached, also has LRU, so you could set a maximum memory size, and it will throw out cold data. That would help overflow. Also it now has extstore so that you can use disk for storage. So theoretically you could store the entire 400m active scraper set if it came down to it. How fast that database performs on-disk is unknown though.
DragonflyDB has lower per-record memory consumption vs memcached. It may be possible to get closer to 175 bytes average per IP address. It can communicate over the memcached protocol. I haven't experimented with it yet.
Lots of room to make it feasible!
Last edited:
Nice - sort of projects I like sinking my teeth into as well
Though even 10 million or 100 million IPs is small if you need to maintain long term historic IP data yourself too.But I'll focus on Cloudflare level for now
For my aggregated Cloudflare analytics dashboard across multiple CF zone sites HTTP/WAF data, I now have 4 sources of IP data from Cloudflare Enterprise IP Intelligence, IPLocate.io, FindIP.net and Proxycheck.io just to see what info available on top threat actor/bot IPs. Each provider can be toggled off. But interesting FindIP.net is surfacing data the others aren't reporting.
Nice - sort of projects I like sinking my teeth into as well
I think we can use MySQL or SQLite as a long term storage engine if we steal a few ideas from Redis on how it writes to disk in the background ( an 'eventual consistency' type design ). Or maybe extstore actually works out.
Really interesting, i wonder how findIP is free.
I find it strange that there's no information about the company itself on the webpage. And socials are dead.
I can't find any gotchas in their terms of service or privacy policy.
Might implement this one, thanks for the tip
Or Cloudflare KV/D1 SQLite/R2 layered approach then can bind to Cloudflare Worker per site as reverse proxy too
Yeah surprised when I found them - maybe get AI to deep dive into them LOL
Last edited:
Nice... but I assume you're atleast on a 4 core system, which would handle that like a boss.
That pay per crawl is interesting... and what was in one of the articles, how Google really isn't a search engine nowadays, as they bake AI results at the top into every SERP returned, let alone what they've done for a decade or more now with paid results first.
That looks interesting... free and unlimited use.
Man... they're not even trying any more...
Literal UA in the logs this morning. Not sure if this is a legit but misconfigured bot... or some sort of new scraper that someone was too lazy to configure.
Code:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko, Mediapartners-Google) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION]Literal UA in the logs this morning. Not sure if this is a legit but misconfigured bot... or some sort of new scraper that someone was too lazy to configure.
Mediapartners-Google makes it a bit interesting, but the IP address says Google: 72.14.199.139.Seems indeed to be genuine Google:
Seems, even they do no longer know what they are doing.
Something bizarre I've started seeing are unauthenticated requests to /posts/N/bookmark and /posts/N/report, it'll randomly ramp up from 100-200 of these per hour to thousands per minute, with only 1-2 requests per IP address from many ASNs across many countries, but mostly Vietnam, and Hong Kong. I can't imagine what possible purpose this would serve since they all return a 403, other than an easy page to hit that doesn't get cached because of the permission error. Ended up blocking these URLs when there is no session cookie at nginx because it was causing problems
And now got another email blast from Cloudflare... seems they are really working to revamp their AI/bot control features. Which might explain why some folks on here noticed some 'bumps on the road' with bots getting past filters this past month.
Which is literally yesterday's news as it refers to the latest blog post we have been discussing yesterday. So absolutely nothing new.
We are also getting hit very badly by these bots and it's affecting our Analytics. In particular, Analytics says a lot of traffic is going to "(other)", "Oops! We ran into some problems." and "Log in". Engagement times for these particular pages are 0-1s and it's really affecting our ad revenue. I can understand for the Log in page, but not when it's as high up as it is on the Analytics chart. We're on the Pro version of Cloudflare and have Super Bot Fight enabled as well as our own many security rules + add-ons etc enabled to mitigate this, but it doesn't seem to be going away. I'm not sure what to do at this point.
Should not be to hard to exclude these pages or simply exclude the indetified bot visits from your performance chart. Advertisers won't pay for bot visits anyway and neither for error page visits and if the bots are unwanted, additional traffic on top of a serious amount of genuine users cutting them out should be no big deal. It is just noise, nothing more. Or am I wrong here?
Based on the very vague information you provided it is impossible to give any useful hints.
Sadly I was made aware by the technical/revenue team at the ad network that a large amount of bot traffic driving fake engagement (spending 0-1 seconds on pages at a high rate) does indeed affect performance, and we hadn't started getting this issue until this year. We get about 1.5m-2.2m pageviews on a typical month, but bots are driving this up to 3m+.
I've been having an occasional protection breakthrough ( UAM not enabled ) on my cloudflare client for a few months. Others are reporting similar.
I think only eva2000 is having a good time ( enterprise plan though )
Mmhm.. i imagine that is a problem... and the solutions are the same.
My google analytics have been cooked for a while.
My clients on Cloudflare Business and Pro plans are doing relatively well too. Helps to have my CF Enterprise plan analytics to get deeper insights to craft WAF rules for non-Enterprise plans like CF Business and Pro.
FYI, with Cloudflare For SAAS feature, you can also extend your plan's coverage to non-plan/non-Cloudflare sites too which I have done on CF Enterprise, Business and Pro plans. So I can put a non-Cloudflare account/site behind my Cloudflare Enterprise plan via SAAS feature https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/
Breakdown of CF For SAAS features supported by CF Plan at https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/plans/
CF Enterprise SAAS is awesome though
Interesting cost avoidance strategy.
From what i could tell, if you can get all your clients rolled into one account, you could split the cost of a plan with multiple clients, no?
From what i could tell, if you can get all your clients rolled into one account, you could split the cost of a plan with multiple clients, no?
Oh yes I don't like paying for bots to use up my site resources. I developed my own plugin using Chatgpt and I have audit all the traffic each day and make recommendations for how we can limit the bots even further. It's working really good. I probably should replace I probably should release the plugin.
