Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Yer... I think mine has about 30k online at the moment. Pretty small compared to previous stuff.
I was actually thinking about this in a future project. I know there are addons out there trying to stop all this stuff automatically, but they all do an average job. They do something, but none fix the real problem. Based on discussion here, residential proxies tend to be new visitors, they like to set a figure, usually under 40 visits, and then they bounce to the next IP. I think that alone can be watched, logged, then auto blocked. Nothing else needs to be done, you just need a small addon to recognise patterns in your log, then take action, whether it be to slow it down or throw a captcha, or bounce it back to CF and let its system handle it.
I have thoughts on it... and I like a very simple system that concentrates on patterns. Grab all the known bots that you are ok with, add their IP ranges, then assess everything outside of that for visit pattern. There are some many variables you can use to match against, just simple stuff, like hitting x 404's, file locations real visitors don't access, etc etc etc. Just keep it simple. No heavy database, just use existing logs to assess and recommend action or take action.
Yer, but I've already demonstrated in this thread that that is easy to handle these bot waves on small servers, just optimise the crap out the server and it fixes the entire problem, doing nothing else. Mines just under 2M, doesn't budge being hit. I think the test forum I used in this thread was near 3M... on 2 cores and 4gb ram, took it like a champ upto 3 million daily uniques.
.... if your problem is "load" and not the fact that the bots scrape your content. You wrote that you don't care about that - but other people do.
Its such a lovely little peak.
Caveat: its your site, your choice, purely just curious.
Legitimate issue: webhosts who are paying for resources, that is an issue for them, its not about content, its about resource abuse.
If it were private, that I understand. If people were creating user accounts to scrape logged in content... now that's a different story. But public? That's like putting something on social media and then getting upset because someone screenshot it and posted it on their social media. You put it out there, its there... you have search engines that provide zero traffic to your site, scraping your content. You have link companies scraping content purely to assess links. You have AI scraping content for training use and to provide search results and suggestions to discussion.
I'm not sold these bot waves from residential proxies are scraping content vs. just being painful people to test things. Example, its hard to DDOS a site nowadays directly... but if you overwhelm the site from all different IP's that look like legit visitors? You achieve the same thing. If this menacing behaviour then pushes people in directions for paid solutions... that is a motive to look like scraping sites.
I'm not seeing this behaviour on business sites and blogs. It just isn't there. This seems very targeted to forums. Are phpbb, vbulletin, invision, forums dealing with the same issue?
Why do you care? If you put your content publicly on the web, then its public. Who cares who is scraping your content when its public?
Caveat: its your site, your choice, purely just curious.
Legitimate issue: webhosts who are paying for resources, that is an issue for them, its not about content, its about resource abuse.
If it were private, that I understand. If people were creating user accounts to scrape logged in content... now that's a different story. But public? That's like putting something on social media and then getting upset because someone screenshot it and posted it on their social media. You put it out there, its there... you have search engines that provide zero traffic to your site, scraping your content. You have link companies scraping content purely to assess links. You have AI scraping content for training use and to provide search results and suggestions to discussion.
I'm not sold these bot waves from residential proxies are scraping content vs. just being painful people to test things. Example, its hard to DDOS a site nowadays directly... but if you overwhelm the site from all different IP's that look like legit visitors? You achieve the same thing. If this menacing behaviour then pushes people in directions for paid solutions... that is a motive to look like scraping sites.
I'm not seeing this behaviour on business sites and blogs. It just isn't there. This seems very targeted to forums. Are phpbb, vbulletin, invision, forums dealing with the same issue?
Many of the visits from residential proxies are one-shot - a single request from a single IP, which makes it very difficult to establish a pattern if you're using the IP address as an indicator. When they have access to "400M+ IPs", they don't need to re-use them for scraping a single site.
I see it on Wordpress blogs, where there's lots of content worth scraping. The scrapers can get a bit trapped in scraping the same info via different tags, categories, dates and the numerous other different ways of finding the same blog posts/events.
You're right, BUT, there will still be a pattern in there. Sites typically have members from certain countries. If 100M IP's are hitting your site for one page, then moving on, that is a pattern. Humans browse with behaviour... its the entire foundation of what website marketing is built upon. Psychology can be applied to this. If you distinguish human behaviour from everything else, you split the pool. There will always be false positives, but again, if you have split your IP pool on a behaviour and offer them up a protection mechanism, then you will also record how many false positives you have and use their analytics to limit false positives.
That is actually pretty interesting. Is it about content depth?
Some of it’s content depth - text and image-dense news-based blog posts, other times we’ve seen bots repeatedly scraping a calendar that has recurring events. Great to know that a genAI somewhere knows what will be on in a pub in South London in 2038!
We put challenges around some of the “list by tag” and search paths to make them back off a bit.
Prior to me implementing harsh limits with Anubis on one particular site (Game combat log parsing website), it's the "inch deep, mile wide" idiom that comes to mind. The site doesn't have too much in depth (page depth), but it's a mile wide with content (users over many years uploading content). The bots/scrapers prior to Anubis being implemented absolutely were ravaging said website. At its peak, it ultimately resulted in pipe saturation (upstream bandwidth) and causing system stability issues (backend machine IIS server and the associated MariaDB daemon) with high CPU and Disk utilization due to the absolutely extreme amount of connections taking place in such short amount of time. Effectively DDoS'd the website.
Thus, I don't think it's much to do with the depth of a website, more or less in the mindset of "bots see link, bots go to link" programming.
The reasons have been told 1000 times, not just by me. In my case: My forum is small but has very high quality content within its special niche. Lots of it is basically exclusive: You cannot find it anywhere else as it has either been created by forum members or it is aggregated in a way that you won't find anywhere else (and would need years to find the sources that are condensed here, if they are still available which many of them are not). All of that used to be free and most of it was accessible w/o limitation or registration. This amount of information and the fact that it is pretty comprehensive as well as pretty reliable led to the fact that my forum is constantly gaining new members, as it is more or less the best source of information in it's niche. Which btw. even the various AIs state if you ask them.
The effect is that my forum is a pretty vital and a flourishing community which also translates over the borders of the net into real life meetings and many members that came for information reasons initially have become part of community and later also of that real life community over time.
If now AIs grab this content for free and deliver it through their interfaces or through search engines AI mode people have no reason to visit the forum any more and over time the community will dry out and die. Also, as my forum is non-commercial and over time countless hours have been spent to create the information pool: Why should I give it away years of work for free to some mobsters, that then sell what they grabbed for free for their own benefit? That grab w/o asking and ruthlessly impact the performance of the forum and so even cost time and money through their behavior?
If that wasn't enough: As in every forum also in my forum people tell a lot of bits about themselves and often are not aware what effects this can have with today's and even more future technology. It has been shown earlier in this thread that it is no big deal to identify people throughout the net with the information bits that they leave as a trail. So it is also a matter of caring for the privacy of forum members to lock AI scrapers out.
The last factor alone should be enough to fight the scrapers, even if your forum has no exclusive content or no content beyond general chitchat.
As a consequence I do fight the scrapers for almost two years now with a steep learning curve and despite the asymmetry in resources available until now with good success as it seems. I have also put most parts of the forum behind the registration wall - some have always been there, but most parts were freely accessible. Now, guests see only the first post of a thread (and sometimes only the thread topic) and the first couple of lines of articles. Sometimes, I deliberately post relevant bits only in the second posting of a thread.
Since I made this change registrations have roughly tripled or quadrupled. I hated it to be forced to this move but in the end it was the only reasonable and rational thing to do.
Last edited:
My custom Cloudflare analytics tool and dashboards also aggregate all configured Cloudflare zone sites' traffic metrics and Bot Crawler identified traffic to give my clients and me actionable WAF expression rules to try. That is passive method of what you're proposing. I am planning to make it more proactive via Cloudflare workers for clients who are ok paying more for such as Cloudflare workers are billable beyond the free tier quota. Otherwise, passive method is a middle ground as I leverage Cloudflare IP Intelligence data too.
Expanded feature set for my Cloudflare custom analytics dashboard to understand these rogue AI/crawler bots for aggregate and individual Cloudflare HTTP/WAF analytics to 7 site zones analysis (1x enterprise + 4x business + 1x pro + 1x free plan) to also include Cloudflare IP Intelligence that can classify IPs i.e. web hosts, ISP, residential IPs etc where I do IP Intelligence lookups via Cloudflare Enterprise account as it has 2,500 IP lookup quota/month vs non-Enterprise plans at only 100 IP lookup quota/month. As I am only looking up top XX IPs and not all and have it cached for lookups, 2,500 IP lookups/month should be enough for now
Cloudflare IP Intelligence lookups for top threats on aggregated zone data analytics.
Double check you aren't exposing the ports publicly.
Folks, Cloudflare has new options to manage AI traffic https://blog.cloudflare.com/content-independence-day-ai-options/
The also have BotBase but that is Enterprise plan only.
Nice on BotBase you can now filter on each bot and associated detection id by clicking on it, much easier
CF WAF event log filtered entries details
New AI bot controls on all CF plans
Last edited:
Well what's interesting is that previously, there was a 100% chance the scrapers would just keep going after being given a love tap.
Maybe it's a combination of not wanting to be identified AND not wasting money.
At this point it's effectively like a software developer omitting installation instructions.
I think the fact that you have to layer something else on top of cloudflare that you've customized already says some things about how effective cloudflare is
( but i'm not surprised by this, and that's why i'm building an 8 layer system.. )
Here's the problem with that.
The person operating the scraper takes a known good list of URLs, then distributes it across 5-7 figures of unique IP addresses.
You may see 1 or 2 hits from an IP, and there's likely no pattern that's identifiable vs regular traffic.
Pattern matching and rate limiting is useful but it will not work against your typical scraper wave anymore.
I have receipts in hand to prove it
Once you've economized the server, your problem is bandwidth costs.
On Amazon., we pay ~$90 per terabyte. On our large scale sites, bots consume many terabytes per month.
.. and that's with them either running cloudflare OR my very mature fail2ban setup.
You're lucky if you have some unlimited bandwidth hosting.. but.. you may find your provider passing their cost on to you eventually.. no way around it.
I see AI scrapers regularly crashing my little 2 core wordpress servers because they got stuck on some loop.
The more complex the website, the more problems we're going to see.
Another one that gets them is file probers.
These guys found out that since i'm rate limiting 404's, if they send 20 concurrent hits, then half of them will land because there's a big latency between when fail2ban sees logs and the time it acts.
I'd be interested to know if you managed to pull it off!
No easy wins like that in my configuration. The only exposed ports are SSH and HTTP/HTTPS
That's closer to what my sole client that uses cloudflare needs. Good move on their part.
Unfortunately we still have to write a few cronjobs because they want even more granular control than that.
Not sure how common a stance that is though.
Last night we peaked at 5.5m unique visitors
It is not exactly a big nothing burger but it is a massive advertisement of their Pay-Per-Crawl marketplace (which does not seem to get any traction, else they probably would have proudly announced that. It is also advertising for their bot directory, which is, in fact, pretty useless until now. What they now want to do is de facto use their market position to force bot providers to certify their bots with cloudflare. I don't say this would be a bad thing - but given the monopoly Cloudflare already has it is a pretty dangerous game.
The big fat nothingburger is that they (again) do not say anything about the swarms of scrapers that come through residential proxies - this is the biggest issue currently and yet, for the second time in a recent article by Cloudflare where they promise to solve the issue they completely ignore the whole topic. This probably says something.
This (quote from the blog article)...
... seems partly misleading. I was monitoring my Google rank when I locked guests (including search engines) out from anything but the first post of a thread and was a bit in fear if my ranking or the traffic from google would crash. Neither was the case. I deliberately let them see that there are new threads and the first post instead of putting the whole thing behind the registration wall and I have a huge message box for guest visitors that guest access is very limited due to scrapers and AI bots. As a consequence, the AI bots report that to the user when it mentions my forum and says for detailed information free registration in the forum is needed. Job done.
In general I like the idea of the bot classification that is outlined in the blog post - however, I do have my doubts that it works and serves anything other than to stabilize CF's monopoly. The idea to again refer to robots.txt seems somewhat ridiculous - for bots that follow robots.txt we don't need Cloudflare and i.e. the combined bot that apple uses can be instructed not to use the content for training for ages already. So CF ignores the existing options and tries to establish a standard for it's own benefit and - even worse - to convince people again that robots.txt would be a valid way to deal with scraping and AI bots if you only use Cloudflare. What a bad idea.
They do however leave out one of the most basic measures: Put in your TOS, in your footer, in robots.txt and where ever you may like it a clear statement that scraping, using the forum content for training of AI models and/or commercial usage of the content is not allowed. This won't stop any scraper but costs you nothing and may bring you into a better position if in future at some point courts have to decide if the scraping was legal. Even every single one of the sellers of residential proxies that claim that their business would be legal state that scraping becomes illegal when the TOS of a webpage forbid it...
Had some shower thoughts today about my protection mechanism.
Proxycheck.io is pretty transparent about how they work.
One thing they do is latency fingerprinting and it sounds like they use honeypots to do it.
It's not a bad approach but, if bot operators catch on, that approach is in trouble.
If in my system, we do this latency fingerprinting on your server, we collect relevant, real world results.
Everyone in your fleet and their fleets you've connected can now collect and use that data to increase the proxy identification accuracy.
We could make this fleet very large if the protection system was delivered as a plugin per platform. Then we can standardize criteria, and increase the trust in the results. We can also make a giga-fleet per application. By doing that, we push accuracy higher.
Obstacles:
Problem: my system requires installation of memcached. This will not work on any shared hosting system.
Solution: i have two leads on possible solutions. One includes a line to the SQLite development team ( or monkeying with a profiler until we make it goo vroom vroom ), the other is finishing a prototype of an in-memory database that runs as a PHP process. But initially, we want big servers in the fleet who typically aren't on shared hosting solutions and have a capable nerd operating them due to scale.
Problem: A very large fleet is gonna produce a heckin' lot of data. Our system may need to catalog a billion IP addresses one day.
Solution: Small datacenter
IDK, i kinda feel like starting another company around this idea.. but i already run two companies.
Feel like i need to round up some business + technical partners, because this is pioneering and difficult work.
If anyone is interested in collaborating on my 8 layer security mechanism idea, please PM me or make a shout out here.
Proxycheck.io is pretty transparent about how they work.
One thing they do is latency fingerprinting and it sounds like they use honeypots to do it.
It's not a bad approach but, if bot operators catch on, that approach is in trouble.
If in my system, we do this latency fingerprinting on your server, we collect relevant, real world results.
Everyone in your fleet and their fleets you've connected can now collect and use that data to increase the proxy identification accuracy.
We could make this fleet very large if the protection system was delivered as a plugin per platform. Then we can standardize criteria, and increase the trust in the results. We can also make a giga-fleet per application. By doing that, we push accuracy higher.
Obstacles:
Problem: my system requires installation of memcached. This will not work on any shared hosting system.
Solution: i have two leads on possible solutions. One includes a line to the SQLite development team ( or monkeying with a profiler until we make it goo vroom vroom ), the other is finishing a prototype of an in-memory database that runs as a PHP process. But initially, we want big servers in the fleet who typically aren't on shared hosting solutions and have a capable nerd operating them due to scale.
Problem: A very large fleet is gonna produce a heckin' lot of data. Our system may need to catalog a billion IP addresses one day.
Solution: Small datacenter
IDK, i kinda feel like starting another company around this idea.. but i already run two companies.
Feel like i need to round up some business + technical partners, because this is pioneering and difficult work.
If anyone is interested in collaborating on my 8 layer security mechanism idea, please PM me or make a shout out here.
Yeah why at Cloudflare layer probably the best solution otherwise resource requirements will grow exponentially.
Re: smallwheels
Yeah i don't really like cloudflare's moves in the last few years. Basically the fate of the internet is in their hands, and they and their investors are showing signs that they plan to take advantage of that.
Even if they weren't doin' capitalism, since they are slowly becoming a part of the internet's infrastructure, government will be given a very easy lever to determines who gets protection and who doesn't. This has already been leveraged.
Hate to say it but if nobody steps up, we could basically lose the non-commercial internet to this.
That's why my system uses proxycheck.io initially and batches getting results when the req/sec is high.
However if the operator can build their own fleet.. or just use the mainstream one..
Then we still have the option to avoid centralization at least.
Yeah i don't really like cloudflare's moves in the last few years. Basically the fate of the internet is in their hands, and they and their investors are showing signs that they plan to take advantage of that.
Even if they weren't doin' capitalism, since they are slowly becoming a part of the internet's infrastructure, government will be given a very easy lever to determines who gets protection and who doesn't. This has already been leveraged.
Hate to say it but if nobody steps up, we could basically lose the non-commercial internet to this.
That's why my system uses proxycheck.io initially and batches getting results when the req/sec is high.
However if the operator can build their own fleet.. or just use the mainstream one..
Then we still have the option to avoid centralization at least.
