Server Management - Ongoing DDOS Attacks

Wesker

Well-known member
Looking for someone who manages servers. Someone who specializes in filtering attacks.

We have cloudflare enabled, full mitigation through this, protection on our server IPs but the attacks still happen and are happening right now. Need someone urgently who really understands XF and how to filter all types of flood / volume connection attacks.
 
Could it be they are using the server's IP address to bypass Cloudflare? I'm no expert and I hope one comes along but I believe you can password protect files in the server. Then it's protected both ends - cloudflare and server ip.
 
Could it be they are using the server's IP address to bypass Cloudflare? I'm no expert and I hope one comes along but I believe you can password protect files in the server. Then it's protected both ends - cloudflare and server ip.

They are not. Already tested. It's happened before yes but for example we rotate IPs and swap some new IPs in same issue. Also the IP address have DDOS protection on them. I've dealt with many DDOS attacks over the years. This is the most frustrating and unsolvable one I have had to date.
 
If the traffic isn't identifiable (ie an agent, or URL fragment, network, or similar) then all you can realistically do is look at raw rate limiting. However normally there is something in the attack that gives it away or at least allows you to cull down a decent proportion of the attack - such as cutting out countries or large netblocks whilst you get a handle on things.

You could look to cull out requests that don't have the XF session cookie present - whilst that will impact legitimate guests I'm assuming the DDOS is not from accounts and is guest traffic. Letting members (who are already logged in) through would at least minimise the impact to the users you presumably care more about!

No cloudflare expert, but there are certainly variables accessible to their firewall to filter on some of the above. I'd start by trying to cut things down as much as you can - then you can at least more sensibly look at your logs and see if there are any patterns you can do a more refined job on.

Best of luck, dealing with attacks is never much fun.
 
Talk to @MattW

 
Talk to @MattW


Messaged. Waiting to see if he can help. One time I did contact him he was too busy with projects many years ago.
 
If the traffic isn't identifiable (ie an agent, or URL fragment, network, or similar) then all you can realistically do is look at raw rate limiting. However normally there is something in the attack that gives it away or at least allows you to cull down a decent proportion of the attack - such as cutting out countries or large netblocks whilst you get a handle on things.

You could look to cull out requests that don't have the XF session cookie present - whilst that will impact legitimate guests I'm assuming the DDOS is not from accounts and is guest traffic. Letting members (who are already logged in) through would at least minimise the impact to the users you presumably care more about!

No cloudflare expert, but there are certainly variables accessible to their firewall to filter on some of the above. I'd start by trying to cut things down as much as you can - then you can at least more sensibly look at your logs and see if there are any patterns you can do a more refined job on.

Best of luck, dealing with attacks is never much fun.

Raw rate limit how? Cloudflare already at it's peak. Host needs to rate limit. I just actually requested if they could setup a rate limit through PATH.

However normally there is something in the attack that gives it away or at least allows you to cull down a decent proportion of the attack - such as cutting out countries or large netblocks whilst you get a handle on things.

I am no expert here. I'm sure there is something I am missing and I'm sure if someone with more experience handled this, they could pinpoint what is going on.

You could look to cull out requests that don't have the XF session cookie present - whilst that will impact legitimate guests I'm assuming the DDOS is not from accounts and is guest traffic. Letting members (who are already logged in) through would at least minimise the impact to the users you presumably care more about!

Would kill all guest traffic

No cloudflare expert, but there are certainly variables accessible to their firewall to filter on some of the above. I'd start by trying to cut things down as much as you can - then you can at least more sensibly look at your logs and see if there are any patterns you can do a more refined job on.

Already did so much. Every other attack I would have mitigated by now but this one I am at a lost.
 
Raw rate limit how? Cloudflare already at it's peak. Host needs to rate limit. I just actually requested if they could setup a rate limit through PATH.
Hopefully you've resolved the attack or it has subsided now.

I was hoping you could perhaps rate limit the connections coming through from cloudflare to your own servers so that they are dealing with a level of traffic that is more appropriate (ie x requests per second). Their documentation can be found at https://developers.cloudflare.com/waf/rate-limiting-rules/ although looking at it now unless you are on their top tiers of support you seem to be restricted to rate limits by IP address. Which is going to be of rather limited use in a DDOS that is using a vast number of source addresses. So apologies that may well not have been much use as a suggestion. Although it's been some years since I properly used any other webserver I imagine most have some rate limiting options like Nginx does. Assuming the attack is all at the application layer (which I guess it would be if coming via Cloudflare) some rate limiting there might help, although unless there is an element you can build a common key on you're again stuck if each request is a fresh one.

Would kill all guest traffic
That was the idea! :) Generally if your servers are under extreme load it can be hard to actually do log analytics and so forth - so gaining control has always been my first goto, once you have some control back you can ease up on the restrictions. A really good DDOS would be totally indistinguishable from legitimate guest traffic so blocking that might be all you could do until they got bored or ran out of money.

Already did so much. Every other attack I would have mitigated by now but this one I am at a lost.
Was Cloudflare's "under attack" of no use then, or was the attack large enough that it made little difference? I guess there are probably ways those launching the attacks bypass the interstitial page now. I'm certainly fascinated since Cloudflare is often sold as the solution for these kind of issues, but we've had a few recent threads where attacks have not been mitigated by it. Granted of course traffic offloading via caching and so forth is of value in addition to attack mitigation.

Did you (or Todo10) manage to get this one under control successfully? I'd be interested if you're able to share any learning since really distributed attacks are everyone's nightmare. FWIW in my idle searching this morning I did stumble on this (not new and quite basic) guide for DOS mitigation using Cloudflare, reading it now...
 
Hopefully you've resolved the attack or it has subsided now.

No. Has not slowed down, same issues.

I was hoping you could perhaps rate limit the connections coming through from cloudflare to your own servers so that they are dealing with a level of traffic that is more appropriate (ie x requests per second).

Already implemented. Not working.

Was Cloudflare's "under attack" of no use then, or was the attack large enough that it made little difference?

It is always of some use but you need to do additional filters if you truly want to mitigate ddos attacks.

I'm certainly fascinated since Cloudflare is often sold as the solution for these kind of issues, but we've had a few recent threads where attacks have not been mitigated by it. Granted of course traffic offloading via caching and so forth is of value in addition to attack mitigation.

We are as well. This attack should have been filtered right away but it's still ongoing.

Did you (or Todo10) manage to get this one under control successfully?

He hasn't responded back.

FWIW in my idle searching this morning I did stumble on this (not new and quite basic) guide for DOS mitigation using Cloudflare, reading it now...

Been using CF for 10+ years. They do a great job but there are some ways to bypass it.
 
He hasn't responded back.
You could try the company directly: https://todo10.com/en They say they have 24/7 support, but they might not respond to business enquiries out of hours of course. Looks like they are in South America going off the telephone code.

MattW looks to be in the UK so they should be "up and about" now, so might be available.

This attack should have been filtered right away but it's still ongoing.
Assuming you are paying for Cloudflare have you tried their own support lines?

Re rate limiting
Already implemented. Not working.
So does that imply that every new request in this DOS is from a fresh IP and the IP addresses are only being cycled back round on a very slow basis - thereby bypassing any IP based rate limiting. You're only option then would really be rate limiting in the webserver layer (unless you have a very expensive Cloudflare plan and if you had I assume you'd be yelling at them now!) on some other field.

I would assume the traffic is also being spread over multiple countries - there are no hotspots or particularly bad ASNs you can cull to reduce things?

Is your site actually down or just struggling? Is your webserver successfully logging - have you tried looking at things like the user agents to see if there is any kind of pattern or maybe a pattern in the URLs requested? I would assume along with the IP addresses they are cycling both of those, but if you are lucky it's through manageable sets (although no reason it couldn't be through hundreds).

Whilst you can't obviously at a glance tell which addresses are the "bad ones" have you checked a random selection to see if they are listed for abuse (eg with someone like https://www.abuseipdb.com/). Maybe if this DDOS network has been used recently you could build a rough blacklist to help mitigate it based on their data?

What kind of request rate are you seeing?
 
Last edited:
Back
Top Bottom