Ultimate DDoS Solution! Nice on the budget too!

Looks like your not using them anymore :)
And switch to Sucuri/Cloudproxy.
Why? :)
Cheaper, wanted to see how they would work out. Seems decent so far, had quite a few issues up front but smoothed them out and I don't think I've had anyone complain in a while. Definitely had a lot more members being blocked from using the site when I switched though and definitely some weird routing issues. Took us about a month to get it fixed.
 
  • Like
Reactions: rdn
I wish I could edit the OP but I'll give a little update here. We've moved away from Sucuri and went back to Cloudflare, this time using a different system administrator. We installed Centminmod, customized it to our needs, and placed CF proxying our domain. Since these changes, my experiences with CF are much improved compared to the first 2 times I gave them a whirl, which I think the success is mostly due to Centminmod and the new sysadmin configuring the server to play nice with CF.

We were on CF Business plan until recent (because they have an SLA saying 100% uptime from DDoS attacks, which they didn't comply with when we were attacked and knocked offline, and then was told it was our fault we haven't made specific changes in the firewall settings). Have since moved down to Pro after making multiple modifications (listed in this post) as we don't need many of the additions Business plan gave over Pro and the additional DDoS protection did nothing, and the only other thing that was beneficial is Railgun. I felt Railgun alone wasn't worth the extra $180/mo (wish we could just purchase that feature for a smaller fee). We stay within the allotted amount of Page Rules and Firewall rules for Pro so we saw this as the best way to reduce overhead and after making the changes to the firewall rules, the Business plan didn't offer any additional protection than what we get on the Pro plan now after these customizations.

While I know it can be improved, we are sitting at 70-75% caching of data, which can also help with DDoS protection. I've gotten pretty good at analyzing Firewall logs, which I didn't know what I was doing in the past when I was first using CF years ago. As such, we've been able to reduce the effects of DDoS attacks to next to nothing to the point where I'm no longer seeing the attack attempts (inb4 an attacker reads this posts and attempts); prior to these modifications to the firewall rules we were getting attacks up to 3x/day, 5-7 days a week.

Other things we've done is:
  • Only allow connections from CF servers/IPs to our server so if for whatever reason our IP address is leaked (say, if the attack is too big for CF to handle, which has happened in the past with us, they push that traffic straight to the server), now all those connections will be refused access to our server, we don't get nullrouted, and we just wait for that CF node to work again for us.
  • We switched from a VPS handling our email to using Amazon SES, scrubbing the headers so the origin server IP isn't leaked. We had noticed on the VPS our emails were blocked or sent to spam due to the IP address being used in the past for spam. We use SES for our transactional and newsletters btw.
  • Analyze the firewall logs each day to every couple days, making modifications to block or allow certain connections, or change priorities of rules, etc. One thing to note, there are some companies out there that are trying to monitor forums to gather data and then sell to customers and these companies send their bots out and can account for up to 50% of your connections on your site (a single one that is scanning my site is around 30% of the connections to my site currently). These bots can take up valuable bandwidth for some...so these companies/bots (i.e. Brandwatch) get a full on BLOCK from us.
    • I've also put countries known for attacks and hacking on a CHALLENGE list, which I then analyze and see if we need to BLOCK anything coming from these countries. The great thing when you start adding these rules is you can see how effective or ineffective they are. For example, my Country Challenge has 7% of those connections actually completing the CAPTCHA challenge. Another one of my lists is at 5% completion of the CAPTCHA. So I'm essentially blocking 93-95% of either bots or attacks from these countries without inconveniencing many actual human browsers.
  • We've effectively halted all the Chinese spam we were getting multiple times a day by placing a country wide block to our registration page (Page Rule). Sorry China.
  • Made changes in the Firewall > Managed Rules > Advanced settings for each of the rule sets on that page. This is something I never realized existed so many of these have actually helped mitigate some of the attacks we were having.
  • Various other things that I would like to keep secret in the event a would-be-attacker ends up reading my posts to figure out what we are doing specifically and attempting to circumvent our security protocols. However, if you are having problems with DDoS and CF isn't handling the attacks like you think they should, feel free to shoot me a PM if you can't figure out the Firewall rules, Advanced settings to change, and reading the logs.
Comparing this current set up to both DDoS Defend (rip) and Sucuri, the DDoS mitigation is roughly the same, if not better now, I can fine tune different aspects better than those other companies since I can see the logs, I'm not getting Chinese spam bots anymore, and it's saving me money each month. The only thing I'm still having problems with is many users are having issues with editing some posts (i.e. a user had put emoji's in their title and was getting errors on editing, once we removed the emojis, the user could edit...something we are looking into but it's baffling at this point).

With this type of set up:
CF = $20/mo
Amazon SES = varies depending on the campaigns we run but if you are a smaller site, there is a free tier and then you pay pennies for every 1,000 emails

This is the cheapest I've been able to get DDoS protection that works great, site being speedy, and users not experiencing many, if any, issues.
 
Last edited:
  • Only allow connections from CF servers/IPs to our server so if for whatever reason our IP address is leaked (say, if the attack is too big for CF to handle, which has happened in the past with us, they push that traffic straight to the server), now all those connections will be refused access to our server, we don't get nullrouted, and we just wait for that CF node to work again for us.

Exactly what I'm doing, Litespeed made it very easy.

  • We've effectively halted all the Chinese spam we were getting multiple times a day by placing a country wide block to our registration page (Page Rule). Sorry China.

Also same, kinda. I block China entirely at the server level, as well as Ukraine, Afghanistan, Russia, Nigeria, Hong Kong, Vietnam, Indonesia and Pakistan.

The only thing I'm still having problems with is many users are having issues with editing some posts (i.e. a user had put emoji's in their title and was getting errors on editing, once we removed the emojis, the user could edit...something we are looking into but it's baffling at this point).

1.webp
 
Also same, kinda. I block China entirely at the server level, as well as Ukraine, Afghanistan, Russia, Nigeria, Hong Kong, Vietnam, Indonesia and Pakistan.
I don't block the traffic, however, I do put all those countries on a CHALLENGE rule. Only 7% pass so I feel good about that rule lol.
I'll have to see if that works for the emoji's within the title issue we were having. It's crazy, they weren't editing the title, they were editing the post and couldn't...but I could for some reason.
 
've gotten pretty good at analyzing Firewall logs
What method do you use to analyze firewall logs? I'm toying around with logflare (CF App) combined with Google Datastudio ATM. I used Deep Log Analyzer previously.

I've recently upgraded to the Business plan but found it to be useless. The 100% uptime guarantee is bogus and their response time is not much faster. Nor is their help very useful. I was already using RailGun. I will be downgrading again.
 
What method do you use to analyze firewall logs? I'm toying around with logflare (CF App) combined with Google Datastudio ATM. I used Deep Log Analyzer previously.
I know we are also chatting privately but for other users (so they don't think I'm ignoring you lol), right now I'm only using the Firewall logs provided on CF's website. Not using anything fancy. There are a lot of steps I take so it's quite hard to put it down in a post. Basically I will utilize the firewall rules and either filter those results out and analyze that data or I will exclude those firewall rules results and analyze that data then. And then it is just tinkering with things here and there. I may have just stumbled upon something huge but waiting to get verification if another admin gets the same results I just got in the last 24 hours.
I've recently upgraded to the Business plan but found it to be useless. The 100% uptime guarantee is bogus and their response time is not much faster. Nor is their help very useful. I was already using RailGun. I will be downgrading again.
Yeah, Business is completely useless if we are talking about DDoS. It did nothing for us whatsoever. The manual changes to the firewall rules is where we got the biggest bang for our buck and I was posting this information here in hopes that it helps someone else save a bunch of money each month if they are primarily looking for DDoS protection and not any of the additional features that Business plan has to offer. Their SLA is complete bs...I should screen cap what they said to me. Royally pissed me off at the time.
 
One of the biggest things people overlook when using Cloudflare is ensuring there isn't any way for an attacker to get your server IP.

This can happen many ways, for example;
  • If you only just switched to Cloudflare there are databases online where attackers can get your previous IP
  • Mail headers usually contain the originating server IP (as well as the separate VPS relay mentioned in the OP, SES doesn't enforce the Received header, as long as you remove it via Postfix or whatever mail agent you're using)
  • Subdomains not being routed via Cloudflare, that are pointing to the server IP (eg ftp.example.org)
  • Features such as Xenforo's HTTPS image proxy and URL unfurling will leak your server IP (unless you setup a proxy)
  • Any addon which supports fetching of user inputed remote content/images (eg ThemeHouse's Covers addon, and many others)
  • It's also good practice to firewall port 443 (and 80 if its open) so only Cloudflare's network can connect to your server via HTTP/HTTPS.
It doesn't matter how effective Cloudflare's protection is, if your server is getting attacked directly.
 
So funny that after I posted the update, I've been hit with large attacks (but no down time!), especially in the last 2 days. Just spent the last couple of hours looking through the logs, adjusting the rules, adding single IP addresses to the block list, and contacting the ISP's and server hosting providers reporting the abuse of their IP addresses. Guess this person will have to buy a whole new botnet after this lol

I should also mention that most of it was blocked by WAF (with some modifications to the settings) and Browser Integrity Checks. This was a 6 min time frame from Jan 31st:
HGNWNHo.png


NC0qlHq.png


If the trend continues, I have about 3.5-6 hours until the next attack. Will likely see the IP Access rules shoot up towards 70% or higher with the next attack if it's similar to the last 2 :poop:
 
  • Wow
Reactions: rdn
Was a little off on my prediction. Was only 1 hour from posting this

IMG_20200202_084154.jpg

Attack registered as being much smaller (each day it seems to be half of what it was the day before). It appears nearly 100% of the attack was either blocked outright or challenged. There was about 0.5% that was allowed through, but I think that's due to a single firewall rule which the allow is intended.
 
I just wanted to update the information a bit. We've had quite a few attacks since my last posting, none have taken us down. Seems the attacks have slowed down quite a bit due to it having 0 effect on the performance of the server.

I would also like to make mention we've implemented fail2ban which has helped automatically mitigate some of the attacks. I know fail2ban has vastly helped @Alfa1, and I'm sure he can give more insights to how well it's worked for his site.
 
Yes, its possible to connect fail2ban to cloudflare API to ban or challenge all users that meet the criteria that you set. This is a live saver when it comes to Layer7 DDoS. Massive attacks are now hardly noticeable because each IP/user is blocked before it can become a problem. If I would not check the stats then I would not notice the attacks.

@eva2000 and @MattW are able to set this up on centminmod.
 
Yup fail2ban when properly configured can help when it is configured to talk to Cloudflare Firewall's API. Though I am finding these days just standalone Cloudflare Firewall is stopping alot of stuff with custom Firewall rules. Though ultimately still need some tuning on origin backend server side to ensure whatever requests that get through the CF Firewall and CF cache layers, don't overwhelm your backend.

1588661340145.webp
 
Despite what many people say about Cloudflare, it has to be one of the best value for money services, period. Do you realise what you get for just $20 a month? The cost of building such an infrastructure is huge (or yuge as Trump would say). I've been using it for two years. I only had downtime once for 2 hours I believe. It sucked and the reason of the downtime was childish but still. It saves me a lot of money because it allowed me to replace my old server with a new one with lower specs and also stop the CDN I was using. There is no easier to manage dashboard and the propagation time is very quick. I've used almost all of the top commercial solutions in regards to this. Trust me, it's the best and most flexible especially for what you pay.

It's not for everyone. For me it has been perfect up to now.
 
I am pretty happy with CloudFlare now, but... it took me a lot of figuring out before I understood the system and could make it work for me. @AzzidReign helped me greatly by sharing his experience and find my way. CloudFlare is much more advanced than it may seem.

Cloudflare support has often been close to useless. Most of the time they post template replies that do not apply to the situation. The most useful thing they have done for me is to explain to me that their $200 a month business plan is pointless. It does promise 100% uptime but that's nonsense.
it's a real petty that CF does not offer ANY XenForo support, integration or firewall functions.
Rate limiting in cloudflare is quite bizarre because they charge for valid traffic and then mislabel the DDoS traffic they let trough and charge it as tens of millions of valid connections in half an hour. This makes rate limiting quite useless. Luckily there is a better solution...

The centmindmod fail2ban - cloudflare API integration is priceless. It's a game changer. I went from dealing with massive attacks every day and night for years, to not having to worry about it at all. Its amazing that I can just chill out in the evening, go to sleep, wake up, work and just go about my day without dealing with attacks continuously. I was even able to scale back the rules in the CF firewall to let more traffic in because F2B will catch most of the attacks and nip it in the bud before it can grow.
Also extremely useful are many of the posts by @eva2000 around the net about cloudflare. (Here, TAZ, centminmod, CF forums, etc)
 
I was even able to scale back the rules in the CF firewall to let more traffic in because F2B will catch most of the attacks and nip it in the bud before it can grow.
Glad to hear you are doing better! Such a stress relief huh? And this is exactly what I've been doing lately. There are still a few I won't let through but been slowly loosening some.
 
Top Bottom