Crazy amount of guests

[BrettC]

We've been running fine since I last posted to this thread but yesterday got hit with a massive attack that resulted in an error 522, and max'd out our resources.

It's interesting you mention this... earlier yesterday, my name servers (DNS) took in a rather notable amount of requests/traffic of A and AAAA records. At first, I thought it was just another reflection attack (they're typical, and RRL's deal with it cleanly), but this go around it was a single datacenter provider known as WebNX, Inc.. This provider's IPv6 range was absolutely slamming my name servers from various/random IPv6 Addresses with what I'd consider blunt-force dictionary/rainbow attacks.

IPv4 you could deal with, but ugh, these huge IPv6 blocks that these providers are being granted are going to lead us back to the IPv4 issue in many decades down the road. I digress... about an hour later, numerous new IP addresses from ranges that were far outside of previous residential proxies and abusive datacenter IP ranges began trying to access port 443 on the successful hits of A/AAAA records.

Didn't knock anything offline in my case. Most of my firewall and filtering configurations nabbed it all before it got out of hand. I could only imagine the result if filtering and WAFs didn't exist... :/

 

[BrettC]

Proxy providers rotate the IPs out and bring in new ones constantly. So it may be a bit of that going on.

This is what I am recently seeing as well. I am seeing more and more IPv6 clients, on top of IPv4, being used in my case.

 

[chillibear]

FWIW ditto on the IPv6. We don't actually run public facing IPv6 (sorry I know we should and I'm satan's sys admin, etc. etc.) but do have some clients with Cloudflare and other systems in front of the services we provide that do offer IPv6 interfaces (and pass back an IPv4 request to ourselves) and the increase in IPv6 scraping and crud over the last few months has been interesting and frustrating to watch (frustrating because we don't often control the frontends passing the data back and and blocking has to be done in application layers because the underlying IPv4 proxy connection is "good" so to speak). It's been rattling about in the back of my mind about what to do when every new request will be a never seen before IPv6 address (although hopefully at least the block will be owned by "botscrapers ltd" or something!)

 

[smallwheels]

it was a single datacenter provider known as WebNX, Inc.. This provider's IPv6 range was absolutely slamming my name servers from various/random IPv6 Addresses with what I'd consider blunt-force dictionary/rainbow attacks.

Could be interesting to contact them and see what they say. They are a US based hoster, according to Crunchbase a small company (11-50 employees) and according to their TOS relatively strict regarding fraudulent behaviour (though your specific case is not covered but possibly indirectly included). However, no one knows - there are a bunch of these companies that have a serious website which in fact just serves as a friendly cover to a bad-to-the-bone-company. But maybe they have to say something.

 

[BrettC]

FWIW ditto on the IPv6. We don't actually run public facing IPv6 (sorry I know we should and I'm satan's sys admin, etc. etc.) but do have some clients with Cloudflare and other systems in front of the services we provide that do offer IPv6 interfaces (and pass back an IPv4 request to ourselves) and the increase in IPv6 scraping and crud over the last few months has been interesting and frustrating to watch (frustrating because we don't often control the frontends passing the data back and and blocking has to be done in application layers because the underlying IPv4 proxy connection is "good" so to speak). It's been rattling about in the back of my mind about what to do when every new request will be a never seen before IPv6 address (although hopefully at least the block will be owned by "botscrapers ltd" or something!)

I implemented IPv6 on my server-stuff about four to five-ish years ago. Most mobile clients (LTE/5G stuff) that contacts my servers are on IPv6 native, and no longer routed thru the IPv4 CGNAT or IPv4 native. Now... while many IPv4 clients still do hit my services, as the years have gone by, IPv6 is really catching on, and IPv4 is starting to look less and less on the logs.

In 2026 thus far, I would assume it's closer to a 60/40 split where 60% being IPv4. In years past, IPv6 was much less for obvious reasons (new tech, bleeding edge, entities slow on the uptake, etc.). From my little piece of the internet... name servers / resolvers are also starting to default to IPv6, with IPv4 fallback on queries.

Despite the massive swaths of IPv6 addresses, it's a transparent implementation for the client. I'd highly recommend implementing IPv6 for your customers/users.

 

[BrettC]

Could be interesting to contact them and see what they say. They are a US based hoster, according to Crunchbase a small company (11-50 employees) and according to their TOS relatively strict regarding fraudulent behaviour (though your specific case is not covered but possibly indirectly included). However, no one knows - there are a bunch of these companies that have a serious website which in fact just serves as a friendly cover to a bad-to-the-bone-company. But maybe they have to say something.

I generally don't email abuse@someprovider.tld (or the responsible party/admin) much any more unless the provider has a significant compromise occurring and affects my web services, or if there is a persistent directed attack. It seemed to be a one off event/incident, but if it transpires again, I probably will give them a poke. A quick rdap query revealed a few points of contact if the need arises.

 

[vape.to]

bots are back....with vengeance

 

[ES Dev Team]

That sucks to hear cloudflare is not working well over there.
Right now xenforo.com is beating my numbers on average with cloudflare. I'm guessing they're using the same configuration as the XF cloud accounts. I do know that under attack mode is off.

 

[ES Dev Team]

On my one client with cloudflare, we just had a huge bot breakthrough.

Does anyone here have a script that periodically checks some sign of health, like concurrent TCP/IP connections, and interact with the cloudflare API to flip the on/off switch for under attack mode?

I tested and it seems like it remembers the previous captcha state for a user, so flicking it on/off rapidly shouldn't disturb users.

 

[Suzanne O]

On my one client with cloudflare, we just had a huge bot breakthrough.

Does anyone here have a script that periodically checks some sign of health, like concurrent TCP/IP connections, and interact with the cloudflare API to flip the on/off switch for under attack mode?

I tested and it seems like it remembers the previous captcha state for a user, so flicking it on/off rapidly shouldn't disturb users.

That's good to hear.
We're having no problems either as Trend Micro has them sorted.

 

[ES Dev Team]

I'm watching a monster scraper wave.
90% of the addresses are making a single hit, and it's kinda slow.

On the machine with cloudflare, when i toggled the switch for a minute, the intense scraper net didn't come back. So i'm thinking they know they're being detected, and put some mechanism in to halt the run if a lot are encountering some protection.

While watching it, i had an idea.
What if my protection mechanism that could send this to a central server:

• all IP addresses in scraper waves, minus ones that completed a captcha ( assuming that at X traffic, we reduce the trust level )
• all banned/failed captcha/ignored captcha IPs
• minus the ones that solved a captcha and had good reputation prior

The central server tallies up a score over the past week of who was the most present during these scraper waves and the high scorers get labeled as suspected proxies - this can be looked up at the time of the first web hit to one of the servers in the fleet to increase the initial suspicion level.

If you have a big fleet, or could link up to other fleets, you could basically run your own IP reputation service and know that the data is good. It may be more accurate than a big service.

If you could hook up all Xenforo sites ( 1000's? ) to form this reputation service, you could probably solve scraping.
It's very statistically unlikely that one IP address would visit some 5% or higher of those sites; so the threshold for identification could be pretty low.

 

[ES Dev Team]

Interesting artifact i'm seeing during these waves is at one point, they'll slam elasticsearch for a brief period.
I have search for guest turned off so i'm not sure how they're accomplishing it.

 

[puterfixer]

they know they're being detected, and put some mechanism in to halt the run if a lot are encountering some protection

Money talks. No client wants to pay for a proxy service which consumes hits from the proxy pool against some domain but does not produce results because the domain went shields up. So it makes sense for the bots to detect when the difficulty/effectiveness of scraping a domain suddenly increases, and either move on to the next domain or put the campaign on hold.

If scraping becomes economically unviable, only idiots and people with money to burn will keep doing it (looking at @eva2000 's graph showing how OpenAI stands out by far among the sources desperately trying to get all data possible).

 

[webbouk]

We've been hit again last night and into this morning, so UAM engaged again.

Is Xenforo doing something different as I noticed theyve switched UAM off and arent exeperiencing issues?

 

[smallwheels]

Is Xenforo doing something different as I noticed theyve switched UAM off and arent exeperiencing issues?

It would indeed be nice if they would share their best practices. They either did some custom configuration work (which would be helpful to share with their customers that suffer from the same bot issues to a massive degree) or maybe they upgraded to a paid CF level that offers better protection.

 

[zeeb0t]

Are you talking about how XenForo "only" has 1.3k guests atm?

 

[smallwheels]

Are you talking about how XenForo "only" has 1.3k guests atm?

In comparison to the numbers over the last months that is basically nothing, though most of those actual guests are probably not genuine users.

 

[zeeb0t]

In comparison to the numbers over the last months that is basically nothing, though most of those actual guests are probably not genuine users.

I use Cloudflare, but also my own add-on - so basically you can pretty much guarantee most of those guests are bots that Cloudflare "trusts" (they are mostly not beneficial for you).

I might go through all my Cloudflare settings and share them later on this evening.

 

[webbouk]

We tend to have approx 500 or so genuine guests in our busy periods. Our VPC can handle 15000, the maximum was 36,000 but its not so the number of guests its the amount of server hits they are making.
From what I've found, the demand on our VPC goes ballistic when the guest count rises above normal. So definitely scraping rsther than viewing.

 

[smallwheels]

I use Cloudflare, but also my own add-on - so basically you can pretty much guarantee most of those guests are bots that Cloudflare "trusts" (they are mostly not beneficial for you).

I still don't use Cloudflare and mitigated my guest numbers from a couple of hundred through the bot waves last year down do 20-80 via massive blocking in IP Thread Monitor. (Before that, I blocked via .htaccess pretty massively already and the most I had was 1.800 through a wave coming from Hetzner last year, which lead to an even stricter blocking approach).

Bot Guard brought this further down and what I see at the moment is pretty typical now:



Life is good.