Monster Bot/DDoS attack overnight.

[GrnEyedDvl]

I will spoiler some images at the bottom of the post.

I wanted to share what happened overnight so people can have some idea of what to expect and come up with possible ways to handle it. I run a dedicated Ubuntu server, 64 cores with a ton of RAM on nginx, and we started getting hit by 100,000 requests per minute or more starting about 930 PM or so Central time US.. We are already behind Cloudflare and have their bot and scrape protection turned on. When I got a note from someone that the site was timing out I jumped on and server load was at nearly 1000, and climbing.

I immediately put Cloudflare in Under Attack mode, and slowed down a little bit and then picked right back up. It eventually peaked at server load just under 1200 at which point I just killed nginx and mysql. I was getting traffic that mostly (come back to this in a second) went through Cloudflare because they were disguising their User-Agent block, changing them every few minutes to look like normal users and they totally ignored robots.txt.

I have thousands of IPs, mostly from Brazil but some from China/Hong Kong and Singapore as well. As near as I can tell about 70ish% of this traffic actually went through the CF protection and 30ish% was direct to my IP. I am not 100% sure on that as I think there was a TON of logs in the buffer that did not get written before I killed nginx and I generally do not log every single access anyways. Either one by itself would have been a major issue, combined the server just could not handle it.

After blocking the entire world outside the US via CF rule so I could get web services up and start looking at things, this is when I noticed the User-Agent changes on identical IPs and that some were not coming from the Cloudflare proxy but were instead coming in direct. That is a problem.

I found a script that updates the UFW firewall rules with Cloudflare IP addresses. Basically what you do is shutdown all services so you can work (web, database, etc) and quickly disable UFW reset all the rules via the ufw reset command, then allow SSH from your IP and enable the firewall again. Now you have a firewall that will not allow anything in at all.

Clone the git repository above, then edit the file. By default it allows all CF IP addresses to access all ports which is a bit insane. There are a couple of options in there, one to allow port 80, one to allow port 443, and one to allow both. Comment all of them out and then uncomment the option for port 443. That is really all that CF needs to pass to your server.

Run the script, you will end up with a UFW ruleset like this. The REDACTED part is my home IP address. Then you can add back in anything you need for FTP or email or cpanel and restrict it to wherever you need to. Then setup a cronjob to run the CF IP script once or twice a day to catch when they update their IP list.

Spoiler: UFW ruleset

     To                         Action      From

     --                         ------      ----

[ 1] 22/tcp                     ALLOW IN    REDACTED              

[ 2] Anywhere                   ALLOW IN    REDACTED        

[ 3] 443                        ALLOW IN    173.245.48.0/20            # Cloudflare IP

[ 4] 443                        ALLOW IN    103.21.244.0/22            # Cloudflare IP

[ 5] 443                        ALLOW IN    103.22.200.0/22            # Cloudflare IP

[ 6] 443                        ALLOW IN    103.31.4.0/22              # Cloudflare IP

[ 7] 443                        ALLOW IN    141.101.64.0/18            # Cloudflare IP

[ 8] 443                        ALLOW IN    108.162.192.0/18           # Cloudflare IP

[ 9] 443                        ALLOW IN    190.93.240.0/20            # Cloudflare IP

[10] 443                        ALLOW IN    188.114.96.0/20            # Cloudflare IP

[11] 443                        ALLOW IN    197.234.240.0/22           # Cloudflare IP

[12] 443                        ALLOW IN    198.41.128.0/17            # Cloudflare IP

[13] 443                        ALLOW IN    162.158.0.0/15             # Cloudflare IP

[14] 443                        ALLOW IN    104.16.0.0/13              # Cloudflare IP

[15] 443                        ALLOW IN    104.24.0.0/14              # Cloudflare IP

[16] 443                        ALLOW IN    172.64.0.0/13              # Cloudflare IP

[17] 443                        ALLOW IN    131.0.72.0/22              # Cloudflare IP

[18] 443                        ALLOW IN    2400:cb00::/32             # Cloudflare IP

[19] 443                        ALLOW IN    2606:4700::/32             # Cloudflare IP

[20] 443                        ALLOW IN    2803:f800::/32             # Cloudflare IP

[21] 443                        ALLOW IN    2405:b500::/32             # Cloudflare IP

[22] 443                        ALLOW IN    2405:8100::/32             # Cloudflare IP

[23] 443                        ALLOW IN    2a06:98c0::/29             # Cloudflare IP

[24] 443                        ALLOW IN    2c0f:f248::/32             # Cloudflare IP

[25] 443 (v6)                   DENY IN     Anywhere (v6)

That solves one problem, the direct access instead of going through CF. For the ones that make it through CF I finally got around to setting up rate limiting in nginx.

In /etc/nginx.conf, inside the http block

limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;

And inside either your /etc/nginx/conf.d/default file or inside your /etc/nginx/sites-available/your.site.vhost file. This is the location block suggested in the XenForo manual. I gave it a burst limit of 15 per second.

        location / {
                     limit_req zone=mylimit burst=15 nodelay;
                    index index.php index.html index.htm;
                    try_files $uri $uri/ /index.php?$uri&$args;
        }

And a couple of shots of how bad this was. After I killed nginx and load started dropping I grabbed a screen shot. And I also have a shot from the ISPConfig monitor that shows data transfers. That is a 5 minute window. That big jump makes normal traffic look like 0.

Hope this helps someone out.

Spoiler: Server load

Spoiler: Data Transfer

 

[GrnEyedDvl]

And as of an hour ago I am getting hit again, rate limiting be damned. I do not know much yet, but look at the load on this thing. 1400. I have never seen a load that high. Not pulling tons of data this time, just opening the connection and sitting there I guess. I am open to suggestions.

 

[Seeker-Smith]

Rate limiting in nginx is a great idea. Thanks for sharing.

 

[Jeremy P]

You can also use authenticated origin pulls to ensure traffic is coming from Cloudflare. Maybe not as efficient as the firewall, but know that their IP addresses can change over time.

If the attackers already know your origin IP, I'd recommend finding out where it is leaking from. Typical culprits are email headers (which we don't have control over), or outbound requests from XenForo (from things like the image proxy). You may want to configure an outbound HTTP proxy, or since you're already using Cloudflare then this add-on can automatically set up Cloudflare workers to do it. Once you've stopped the leak you may want to rotate your server IP address.

 

[MySiteGuy]

Reduce keep-alive time, and the number of keep-alive requests a connection can send before Nginx closes the connection. You may also want to consider turning off keep-alive completely.

Reduce connect time out as well.

 

[MentaL]

Are there any specifics in terms of what is being accessed? like URL?=Varient or similar? If the main brunt is one geo, I would block that GEO temp until it slows down. Find a common factor used due to the attack.

 

[GrnEyedDvl]

Rate limiting in nginx is a great idea. Thanks for sharing.

It didn't work as well as I thought it would. I came up with a different way to do it that works much better. My original way had a few problems, but I am still learning nginx. I have only neen using it for a couple of weeks now after being on lighttpd for years.

• It wasn't a global solution, meaning it only applied to that vhost, that is the only host on this machine but still...
• It was tied to location / { which means location /threads { and location /members { and all the rest needed that block too.
• The vhost file can get overwritten with the default vhost file if you aren't paying attention in ISPConfig

Now my /etc/nginx/nginx.conf file looks like this and its been a few hours since I enabled it and I am not swimming in traffic yet. Then you can remove the location block and this applies to everything on that server. If you want more than a single zone then create zone =two and set that inside location blocks on vhosts and I think that will work. Not entirely sure though as I have not tested it.

http {
limit_req_zone $binary_remote_addr zone=one:20m rate=3000r/s;
limit_req zone=one nodelay;
...

I know that 3000 requests per second seems like a LOT, but its really not considering how XenForo seems to work. I may need @Jeremy P to confirm this but it seems like there are a ton of AJAX calls and others that send SSL requests that count against your limit. When I had that set to a few hundred I would only get a partial page load before stuff just shut down. When I set it to 1000 I couldn't load a thread with lot of posts and images. Especially using the @ feature to mention someone or using the search, or even the Elastic Search feature that shows already existing threads when you start a new one. Its a 64 core server so when load is light its probably spreading things out which inflates the request numbers. I am not sure.

Setting at 3k I can still hit the limit if I refresh a few times but it hasn't interfered with what I would call normal browsing yet. A better solution than the one I came up with might be to limit connections, not requests per second. Or maybe a combination. I haven't tried that yet either.

You can also use authenticated origin pulls to ensure traffic is coming from Cloudflare.

That's a bit later than I want to catch them. If they are being processed by XenForo and a plugin the damage is already done. I have the realip_module installed and it works well.

Maybe not as efficient as the firewall, but know that their IP addresses can change over time.

Yeah but not too often. I have been on CF for something like 15 years and its pretty rare. In the IPv4 space anyways. There just aren't many more IPs to go around so people stick with what they have.

The cronjob to update that script is supposed to take care of that but again I cannot confirm that yet.

You may want to configure an outbound HTTP proxy, or since you're already using Cloudflare then this add-on can automatically set up Cloudflare workers to do it.

That might be worth looking at. Thanks!

Reduce keep-alive time, and the number of keep-alive requests a connection can send before Nginx closes the connection. You may also want to consider turning off keep-alive completely.

I will look at that. Thanks!

Reduce connect time out as well.

Do you have a number in mind? I do not know what the default is but I will be looking it up.

Are there any specifics in terms of what is being accessed? like URL?=Varient or similar? If the main brunt is one geo, I would block that GEO temp until it slows down. Find a common factor used due to the attack.

Basically the entire site of 14-15 million posts. Some threads that have not been posted in since 2007 or 8. And some created yesterday. It really feels like a bot scrape, but from all over the place. A month or so ago we had a huge number mostly from Brazil and Venezuela. Same thing last night but add in China and Hong Kong and Singapore and Romania and probably a bunch of others I did not catch or take the time to look up.


And now for perhaps the craziest part of this. I have thousands and thousands of these in the access logs from at least 20 IPs.

205.210.31.67 - - [11/May/2025:09:07:08 +0000] "GET / HTTP/1.1" 403 134 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"

That is what they are inserting as the User-Agent in their http requests, and its obscene. And to top if off they completely ignore robots.txt. This is a monster company not some startup with a bunch of college dropouts. And I have now lost all respect for them and in fact will be telling people to avoid them. The fact that they would ignore the standards that they helped put in place steams me almost beyond my ability to post it here without getting banned.

Maybe they misconfigured something and are sending out more than they intend to on the very weekend I am getting slammed. But it should never be configured that way anyways and they know it. I bet if you look in your logs, you find similar.

 

[MentaL]

Is your guest browsing cached? That reduces a lot of load in itself too. I think there are means to optimize it some what but ,with nginx , I have zero experience.

Block those ip and ranges too. Block any agents that follow a pattern but guest caching reduces the load off mysql which hammers the load. If your using CF, use digitalpoints addon to set the permissions and guest caching perms accordingly. Even with under attack disabled its useful for these scenarios

Litespeed Is really good at layer 7 based attack filtration too. Couple with releem for mysql Optimization.

Question, is the database and webserver on the build, dedicated? If so, virtualizing that build to a offset web to its own instance as well as mysql will allow fine tuning off configurations and resource management especially if load management isn't something that is available at an industrial scale. Sadly these things are reactive.

I've got elastic serving on the webserver vm and mysql on its own. Use lan ips for connectivity and that runs smooth. At least we can limit one point of contact more so than tanking the entire server if we need emergency access. You're welcome to inbox me on specifics. But it deffo sounds like data has been scrapped and is being called. Most stresses follow a single pattern, domain-url-?variedstring

 

[Jeremy P]

I know that 3000 requests per second seems like a LOT, but its really not considering how XenForo seems to work. I may need @Jeremy P to confirm this but it seems like there are a ton of AJAX calls and others that send SSL requests that count against your limit.

It's a bit tricky since it varies by site, web server configuration, visitor, and page.

A cold load of this thread for me is ~48 requests (the document, 10 CSS files, 18 JS files, 18 images, 1 PWA manifest) at ~500kb over the wire. We reduce bundling to keep files small and improve cache hit rates now that HTTP multiplexing is common. User-initiated actions or the session keep-alive mechanism may initiate additional fetch (AJAX) requests. A hot load of this thread for me is only 1 request (the document) at ~44kb over the wire. Most user embedded content is lazy-loaded, but it can add additional requests when using the image proxy.

That all being said, if you can prevent the origin IP from leaking then Cloudflare should be able to absorb most/all of the attack traffic so the requests will never make it to your server at all and rate-limiting becomes moot.

That's a bit later than I want to catch them. If they are being processed by XenForo and a plugin the damage is already done. I have the realip_module installed and it works well.

Authenticated origin pulls are handled by the web server (my first link was for Cloudflare documentation, not a XenForo plugin), so later than the firewall but before they get passed to XenForo. In any case, there's nothing wrong with a firewall and cron either

However the plugin I linked later can also enable guest page caching on the Cloudflare edge, so any traffic from logged out users won't hit your server at all.

 

[MentaL]

That all being said, if you can prevent the origin IP from leaking then Cloudflare should be able to absorb most/all of the attack traffic and the requests will never make it to your server at all and rate-limiting becomes moot.

He would need to route the primary ip access to the domain nginx direct ip access and maybe either modify his local mail server headers to remove the ip leak or use a third party like amazon ses. Personally, I would recommend SES. Couple with the DP workers that acts like a proxy, leaks are pretty much null and void. If the ip is leaked and your being targeted directly, time to switch ip to a different range. At least from my experience.

Don't have any subdomains unproxied linking to the new ip though.

 

[Jeremy P]

Yes, that's largely my personal recommendation. You'll gain most of the benefit of Cloudflare by having your origin IP unknown, and using the plugin to enable edge caching where possible (static files, and guest page views if the trade-offs are acceptable) is a big help too. The only traffic that would make it to your server are cold caches and page views from logged-in visitors, and attackers would not know the IP address to be able to circumvent it.

This drastically reduces traffic in most cases.

 

[MySiteGuy]

I will look at that. Thanks!

For a normal siuation, there shouldn't be many cases where more than 5 seconds is needed because a browser is not going to be taking that long to fetch additional items on a page. Dial it down to 2 seconds it you need to.

Do you have a number in mind? I do not know what the default is but I will be looking it up.

Default is 60 which is a very long time for Nginx to wait for a browser to send the request after opening the connection. You can easily get away with half that much.

 

[GrnEyedDvl]

It's a bit tricky since it varies by site, web server configuration, visitor, and page.

A cold load of this thread for me is ~48 requests (the document, 10 CSS files, 18 JS files, 18 images, 1 PWA manifest) and ~500kb over the wire. We reduce bundling to keep files small and improve cache hit rates now that HTTP multiplexing is common. User-initiated actions or the session keep-alive mechanism may initiate additional fetch (AJAX) requests. A hot load of this thread for me is only 1 request (the document) at ~44kb. Most user embedded content is lazy-loaded, but it can add additional requests when using the image proxy.

This is about what I expected to hear and I knew it wasn't simple. But I knew requests was way more than page views on average. Thanks!

That all being said, if you can prevent the origin IP from leaking then Cloudflare should be able to absorb most/all of the attack traffic and the requests will never make it to your server at all and rate-limiting becomes moot.

Well that is one reason for only allowing requests from CF via ufw. Beyond that the assumption that it was a targeted attack just might not be valid. Maybe it was, but its also possible they just found a server that responded to an http request and decided to go fishing.

Authenticated origin pulls are handled by the web server (my first link was for Cloudflare documentation, not a XenForo plugin), so later than the firewall but before they get passed to XenForo. In any case, there's nothing wrong with a firewall and cron either

I did not mean to ignore that link or part of your post. Sorry! Do you guys use that on this site? I guess I have never even bothered to ping you!

However the plugin I linked later can also enable guest page caching on the Cloudflare edge, so any traffic from logged out users won't hit your server at all either.

I have been reading through the stuff on that plugin. Its definitely interesting. I have used some DP stuff before, specifically sphinxsearch for vBulletin 4, so I will be testing that one soon.

He would need to route the primary ip access to the domain nginx direct ip access and maybe either modify his local mail server headers to remove the ip leak or use a third party like amazon ses.

I didnt know Amazon had that service. I was looking at Google a few weeks ago. Amazon sounds like a better idea.

Thanks guys.

 

[Jeremy P]

I did not mean to ignore that link or part of your post. Sorry! Do you guys use that on this site? I guess I have never even bothered to ping you!

To be honest, I'm not quite sure. I do use it elsewhere, but the firewall approach is fine too and largely accomplishes the same thing.

I didnt know Amazon had that service. I was looking at Google a few weeks ago. Amazon sounds like a better idea.

I use SES personally as well. Like most AWS things it can be a bit frustrating to set up, but once you do it's pretty cheap, works fine with XenForo (via SMTP), and doesn't include your origin IP in outgoing emails.