Crazy amount of guests

[ES Dev Team]

A bit more about how Bright Data (previously known as Luminati) are using common consumer devices, specifically Smart TVs and set top boxes) as residential proxies by getting game developers to include their proxy libraries in their apps.

Free Samsung and LG Smart TV Apps Reportedly Exploit Devices for AI Proxy Traffic

Free apps available on Samsung, LG, Roku, and other connected TV (CTV) platforms are quietly enrolling users' smart televisions.

gbhackers.com

I did try to decipher their docs on Bright Shield

It does look a bit like a "protection scheme" - pay us (a lot of) money and we'll limit how much our proxies hit your site, but I might be misreading it.

I looked this company up yesterday and found nothing that can defend against it.
I also found Meta and X had both tried to sue them in 2024, and failed.

This is one of those Israeli companies that operates in a criminal fashion yet seems bulletproof to international law.
So we can rule out the idea of government intervening.

 

[smallwheels]

Yep... one of a few reasons why it's ideal to have the protection inside the application. More information is available inside.
I also think the cookie shouldn't just pass people.. it should reduce the threat score though.

Probably the optimum is an (at least) two step approach where obvious crap is filtered out early (preferably not on the host your forum runs on, so basically a kind of gateway protection) and the finer granular checks that are context and behavior sensitive in a way that depends from the forum application itself are then done at the border of the application or on or very near to the forum host. This should avoid overloading the host (as way less connections and requests will hit it directly) and it is way easier to scale a gateway than an application. In the end you need both (and if you want to speed up thing through caching or a CDN protect against DDOS maybe even more).

The sad thing is that protection against scrapers has to a degree become more time consuming and possibly even more expensive than running the actual forum.

 

[MySiteGuy]

Consider also that a typical visitor's browser will launch tens or >100 connections just for viewing just one page - one is the HTML of the page, the others are for CSS files, javascript files, any embedded images or anything else delivered from the server. So 64 thousand connections could realistically mean just a few thousand real visitors, before your server is overloaded. The options are then to trim down on the illegitimate requests and/or to add more servers behind a load balancer so that overall capacity is now in multiple of 64 thousand parallel connections. But that's going beyond the typical hosting category

If the browser is using HTTP1.x which is under 2% these days, assuming your stack is utilizing HTTP/2+, then it's only 1 connection per browser.

Its the non-browsers where I see a high percentage of HTTP1.x connections.

I'm seeing all my connections here using HTTP/2, so all those resources are actually 1 multiplexed connection.

 

[ES Dev Team]

Probably the optimum is an (at least) two step approach where obvious crap is filtered out early (preferably not on the host your forum runs on, so basically a kind of gateway protection) and the finer granular checks that are context and behavior sensitive in a way that depends from the forum application itself are then done at the border of the application or on or very near to the forum host. This should avoid overloading the host (as way less connections and requests will hit it directly) and it is way easier to scale a gateway than an application. In the end you need both (and if you want to speed up thing through caching or a CDN protect against DDOS maybe even more).

I think if we use memcached, as long as we have a 4 core system, we can keep up with the worst of bot floods onsite.
It's 0.0175ms per read/write operation, and on average we need 3 read/writes. The overhead can be very low.

We can also refer bans up the firewall chain to your hosting provider of choice. Or just use IPTables if you don't have a programmable firewall. This way we are offloading the mechanism quite a bit.

The sad thing is that protection against scrapers has to a degree become more time consuming and possibly even more expensive than running the actual forum.

Yes, i think this ate a year of my time in total that i could have used improving the forum experience and i'm pretty mad about it.

 

[Suzanne O]

I used to get 10 pages long of guests until i started range banning the bots ip addresses.
Now i get as little as 30 guests on my page.

 

[Anthony Parsons]

Yes, i think this ate a year of my time in total that i could have used improving the forum experience and i'm pretty mad about it.

That is the exact real issue. Are you better to chase this and fight it? Or just use a VPS with decent RAM and let it run wild? Option 2 causes me no time in managing this issue nowadays. I tried option 1 and quickly changed to option 2. I check stats every week or so, but otherwise, I don't really care, as they could throw a million requests a minute at my server and it will be just fine. Everything important resides in RAM and caching. My time is much more valuable and productive than a couple bucks a day to solve the issue. You can't get your time back. Now if you have nothing better to do and this stuff makes you happy, even though frustrates you, then sure... option 1 and playing the game may be satisfying to you.

 

[ES Dev Team]

Me.. i'm fighting it. I have 35 servers to manage. it's a fleet wide problem. If i was to add up the bandwidth my clients were paying for collectively, it'd be in the thousands of dollars per month. or 5 figures a year. They'd be happy to collectively fund something that pays off quick.

If nobody fights it, we lose the indie internet. The cheap VPS will no longer be cheap when they start thinking about their bandwidth bills across their fleet.

 

[chillibear]

Can we trust any accesses from Zscaler? They claim to be some sort of corporate VPN/security service.

We have several corporate very much above board clients who use it. However as with any large scale system (looking at you Apple privacy guard) there may well be bad systems "on the inside", most likely through compromises I'd think. I'm still amazed at the number of serious corporations that take a "safe inside the castle" approach to their IT despite having thousands of members of staff. So we generally have to have their (Zscalar) IP ranges whitelisted on our systems.

scraping via residential proxies does (in opposite to scraping from data centers) a single page view (plus most of the time only using fewer requests for the same page than a real user) and then rotates the IP.

That's what we see; on a single site we might only see one or two requests from an IP. We still track and block them because they'll no doubt be bothering one of our other clients a few hours later and we might as well kill off what we can.

Having everything fairly well tuned generally keeps things ticking over okay even with the bot traffic - although I've had to bump up memory allocations on the firewalls!

The odd thing we've been seeing this week is traffic that is just 404ing because it's hitting completely wrong URLs or traffic with odd URLs which look like they "should be" an exploit or something. So loads of requests like:

/auth.logout/text/auth.logout/facebook/logout
/text/auth.logout/text/javascript/text/javascript
/javascript/view_as/text/grid/list/page/1//view_as/list/page/1
/auth.login/text/javascript/facebook/auth.login

And variations on those, none of which were valid patterns. They were easy enough to pick up with some pattern matching and deal with before they hit the application layer and honeypot the IPs, but it did lead to some speculation as to what it was (all residential proxy sources) - an AI Agent maybe? Anyhow total waste of everyone's CPUs and not heavy enough to actually be a serious DOS.

 

[eva2000]

That is the exact real issue. Are you better to chase this and fight it? Or just use a VPS with decent RAM and let it run wild? Option 2 causes me no time in managing this issue nowadays. I tried option 1 and quickly changed to option 2. I check stats every week or so, but otherwise, I don't really care, as they could throw a million requests a minute at my server and it will be just fine. Everything important resides in RAM and caching. My time is much more valuable and productive than a couple bucks a day to solve the issue. You can't get your time back. Now if you have nothing better to do and this stuff makes you happy, even though frustrates you, then sure... option 1 and playing the game may be satisfying to you.

My clients don't spend time on this, they hire me to handle it

Solutions are a combination of Cloudflare + server side optimisations (provided correct sizing choices). They had no problems handling 70,000 to 100,000 guests when they did hit but just tweaking their Cloudflare WAF settings in their CF Pro/Biz plan accounts and they're back down to <10K guests now.

I also use my own forums and Cloudflare Enterprise plan account and traffic/WAF and Enterprise Bot Management analytics and learnings to further improve my clients WAF rules on their lower CF Pro/Biz plans as Enterprise plan analytics can provide more detailed insights

I don't really care, as they could throw a million requests a minute at my server and it will be just fine. Everything important resides in RAM and caching.

+1 that helps alot!

The odd thing we've been seeing this week is traffic that is just 404ing because it's hitting completely wrong URLs or traffic with odd URLs which look like they "should be" an exploit or something. So loads of requests like:

Yeah I get alot of these, with Cloudflare Enterprise plan you can drill down an increment rate limit rules based on response status codes so strict rate limit on 401, 403, 404 status codes as an example

 

[Anthony Parsons]

Me.. i'm fighting it. I have 35 servers to manage. it's a fleet wide problem.

That is a valid reason... you're above the xf admin running their site.

My clients don't spend time on this, they hire me to handle it

Even better.

 

[Rhody]

I have been struggling with this lately, too. I only have 2 Xenforo forums. Bot load has been surging up to 17,000 at a time, and sql server briefly crashes a few times per day.

What I did over this past weekend:
-Installed CLOUDFLARE
-Added XF BOT GUARD by Zeebot and many of his excellent configuration suggestions
-Added App for Cloudflare by Digitalpoint

It has only been 18 hours, but the server has a MUCH lighter load and hasn't had a single crash since. Cloudflare and XF Bot Guard really help!

 

[Levina]

For google, i whitelist IP ranges. I think anyone is making a big mistake by whitelisting a user agent string.

I just found that out the hard way as I suddenly have a ton of so called Googlebots visiting except their IP addresses tell me otherwise.

 

[Chromaniac]

yeah get botguard a try. it whitelist legit bots based on ip and not user-agent. good addon.

 

[smallwheels]

it whitelist legit bots based on ip and not user-agent

Not really. It does rDNS and other checks for some Search Engine Bots but also includes the User Agent. So simply way more sorrowfully than just the transmitted (and possibly faked) user agent. The same what IP Threat monitor (that Levina had in the past) does with a somewhat broader range of Search Engine Bots.

Also, I would recommend not rushing through tools w/o having the used tools well configured before. Levine went from nothing to IP Threat Monitor, switched from there to Cloudflare after only a couple of days, not long ago and shoot herself in the foot with Cloudflare just yesterday, if I remember correctly. Another day, another tool is not a strategy that promises success. I do like Bot Guard, but I know what it can do and what it can't do. A properly working environment with full understanding of the existing toolchain should be a given before adding the next tool. Not sure if this is already the case.

 

[Wildcat Media]

I have a Zscaler dilemma. We are working with one forum member who visits through Zscaler and was blocked by one of my rules. Thankfully I could filter on the fresh Ray ID he sent to us so I could find his request. In that rule, I block a few unwanted bot categories; Zscaler is not among CF's known and listed bots:

https://radar.cloudflare.com/bots/directory?kind=all

And this is also where I have my collection of leftover user agents, and his user agent string is nowhere close to any of these either.

So it's a mystery as to why this rule blocked his access. I changed this to a managed challenge, so at least he can get in.

I think anyone is making a big mistake by whitelisting a user agent string.

I learned years ago...don't trust anything that can be easily spoofed. For me, user agent strings are an afterthought, something I'll tag on at the end to catch any stragglers that my other rules don't get (like I do in my rule above--it's mostly a holdover from my days of using htaccess for filtering). It still catches a stray few, but not many.

 

[ES Dev Team]

Scrapers have been pretty active today on my system.

It looks like fail2ban had some kind of faulty state after months of continuous usage without any tuning, and the server got it's ass kicked.
banned a few new alibaba/tencent ASNs.
Restarting fail2ban afterwards took the traffic down ~75%



Situation stable.

The distribution level of the residential proxies is enormous.
I count an average of 200 tcp/ip connections yet i have 10,000+ guests at the moment.

I hand fed some data into proxycheck.io and it only detects ~50% of the residential proxies. It does a good job finding datacenters.

A behavioral filter would do wonders on Xenforo sites.

 

[Anthony Parsons]

A behavioral filter would do wonders on Xenforo sites.

Now you're talking...

 

[ES Dev Team]

Now you're talking...

It's in my design. The cost is ~1kb of ram per IP to track history and act on it. Expensive but doable.
Bad sequences of page hits and bad first hits for an IP address should be definable.
These would trigger a captcha.

Is it possible to configure such things in cloudflare?

 

[Pepe2012]

I don't know about others but I do have a bit of a dilemma when it comes to these AI scrapers. I use ChatGPT regularly. To help me with texts, to help me with code. And to then deny it access to my forum feels, as I said above, hypocritical. But where to draw the line? Which ones to allow, which ones to block? I have no idea.

This is where I drew the line. I just set up a cloudflare "pro" account. Not easy to set up though.

 

[Pepe2012]

I used to get 10 pages long of guests until i started range banning the bots ip addresses.
Now i get as little as 30 guests on my page.

How do you get the bot IP address?