Crazy amount of guests

[Anthony Parsons]

I have 1 core 1GB right now on a shared plan with HawkHost. It's gone up to about $86/yr from $75/yr and i keep hitting the max usage of the core & ram hits nearly 100% too......so i have my doubts about that.

I will premise this with - I'm not a server tech, just been doing this stuff for close to 30 years privately. Purely my opinion, and I hope server techs here can steer this if needed.

---

Is this because your system is using swap? If so, possibly why your CPU is max'ed out. I would also ask, is that just a NGINX setup? Or is there a user interface involved, like cPanel? The first one is lightweight, the second adds overhead you just don't need. There are other factors I don't know about obviously, like what addons you're using, are you using the latest MariaDB version and is it tuned to your setup, etc etc.

I have 12 sites on my server, I only have 6 cores because its cheaper to have that option than having 2 cores with 16GB RAM and the HDD I need. In all honesty, mine could easily run on 2 cores, and I have a lot more going on than your one site does. I need the RAM to get everything into memory. I shifted away from Linode for this reason, as all they offered me was package increases, paying exorbitant amounts for stuff I didn't need. Linode 16GB RAM is nearly USD$100 monthly. I found BinaryLane here in Australia which allow tailoring your memory and storage, making it super cheap. To illustrate what I was talking about, I am running at about 12GB used RAM, so its cheaper for me to pick the 6 core package on my provider because if I choose less cores and up the RAM, it costs me more than the default package for 6 cores. I actually have my storage at 100GB which brings my monthly cost to $52AUD. Mine will easily run on 2 cores, but I need the RAM and HDD, and that 2 core package with 16GB RAM and 100GB HDD ends up at $70 a month. Sometimes packages are better for price, sometimes not.

Yours should easily run on one core with 2GB RAM. Don't add swap. If caching is setup correct at CF, that takes load from your server. CF caching should be at about 60% for XF, if not, then you have gains available to you right there. If your DB is tuned and you aren't running poorly coded addons, that will remove load.

 

[DarkGizmo]

I will premise this with - I'm not a server tech, just been doing this stuff for close to 30 years privately. Purely my opinion, and I hope server techs here can steer this if needed.

---

Is this because your system is using swap? If so, possibly why your CPU is max'ed out. I would also ask, is that just a NGINX setup? Or is there a user interface involved, like cPanel? The first one is lightweight, the second adds overhead you just don't need. There are other factors I don't know about obviously, like what addons you're using, are you using the latest MariaDB version and is it tuned to your setup, etc etc.

I have 12 sites on my server, I only have 6 cores because its cheaper to have that option than having 2 cores with 16GB RAM and the HDD I need. In all honesty, mine could easily run on 2 cores, and I have a lot more going on than your one site does. I need the RAM to get everything into memory. I shifted away from Linode for this reason, as all they offered me was package increases, paying exorbitant amounts for stuff I didn't need. Linode 16GB RAM is nearly USD$100 monthly. I found BinaryLane here in Australia which allow tailoring your memory and storage, making it super cheap. To illustrate what I was talking about, I am running at about 12GB used RAM, so its cheaper for me to pick the 6 core package on my provider because if I choose less cores and up the RAM, it costs me more than the default package for 6 cores. I actually have my storage at 100GB which brings my monthly cost to $52AUD. Mine will easily run on 2 cores, but I need the RAM and HDD, and that 2 core package with 16GB RAM and 100GB HDD ends up at $70 a month. Sometimes packages are better for price, sometimes not.

Yours should easily run on one core with 2GB RAM. Don't add swap. If caching is setup correct at CF, that takes load from your server. CF caching should be at about 60% for XF, if not, then you have gains available to you right there. If your DB is tuned and you aren't running poorly coded addons, that will remove load.

View attachment 337382

I do have Cpanel, yes, like I said, it's a shared hosting plan from Hawkhost (their "Professional" plan) here.

I'll have to see what can be done..... my idea was to upgrade my account to maybe their "Turbo" option but i'm not sure if that'd be my best option. My hosting for the year now under their "Professional" plan will be $86/yr (not monthly, yearly) so i'm expecting that to go up by a bit if/when I upgrade, but paying something like $70/monthly just isn't in the cards for me right now.

 

[smallwheels]

I just got hit again and my site is inaccessible. I need to upgrade to a VPS sooner rather than later I think.

it's a shared hosting plan from Hawkhost

I'll have to see what can be done..... my idea was to upgrade my account to maybe their "Turbo" option but i'm not sure if that'd be my best option.

Depending from the size of your forum and the geographical distribution of your audience the IP Threat Monitor Addon for XF may also be a solution, either in combination or w/o CF. So possibly no need to upgrade your hosting.

 

[Wildcat Media]

I do know that shared hosting can be difficult--I used it for many years on smaller sites (prior to the "cloud" era), and we were always at the mercy of our server "neighbors." While our sites were fine, some of the other accounts could really bog down the entire server and leave everyone else dragging.

Our host did have strict limits in place, and we were always well within those. Yet all it took was a neighbor getting a rush of visitors, or using an application that used too many resources, and we would all suffer.

I remember we moved twice. The first time was due to our server neighbor(s). The second time, we moved to a larger "virtual" server that worked fantastic for our sites. (We moved mainly because my busiest client was using Magento.) Eventually that client retired after nearly 20 successful years online, and I ended up moving sites to a cloud host where I have root access and aside from traffic hiccups, everything has been working fine. No server neighbors to worry about, like there was on shared hosting.

 

[TMMAC]

So after about half a year since this thread was created, is there a consensus about what works to help prevent multiple thousands of bots?

CloudFlare managed challenge on all guests. I'm interested in Anubis as a DIY solution, but for now happy with CloudFlare free plan.

 

[Anthony Parsons]

I do have Cpanel, yes, like I said, it's a shared hosting plan from Hawkhost (their "Professional" plan) here.

I'll have to see what can be done..... my idea was to upgrade my account to maybe their "Turbo" option but i'm not sure if that'd be my best option. My hosting for the year now under their "Professional" plan will be $86/yr (not monthly, yearly) so i'm expecting that to go up by a bit if/when I upgrade, but paying something like $70/monthly just isn't in the cards for me right now.

You might just need to go back and ask your host questions... you're saying 1 core 1GB ram, but their plans don't offer that on shared hosting. You might be getting ripped here from an older plan they haven't upgraded. All their plans say more than 1GB Ram.

 

[Anthony Parsons]

CloudFlare managed challenge on all guests. I'm interested in Anubis as a DIY solution, but for now happy with CloudFlare free plan.

Anubis is great if you're not using CF, they're both reverse proxies. You really only need one. CF though sits at your DNS, so anything you block at CF uses zero of your server resources, BUT, Anubis, sits on your server, and whilst it reduces server load significantly, attacks still use some of your server resources until blocked. CF also handles caching, SSL, the list goes on and on.

 

[DarkGizmo]

You might just need to go back and ask your host questions... you're saying 1 core 1GB ram, but their plans don't offer that on shared hosting. You might be getting ripped here from an older plan they haven't upgraded. All their plans say more than 1GB Ram.

Click the "Unlimited Plans" link, it's small but there on the same page, i'm on the "professional" plan.

but I agree, it does seem like an outdated plan and those prices for the current ones? they won't give me those prices cuz i'm not a 'new' customer.

 

[Wildcat Media]

CF though sits at your DNS, so anything you block at CF uses zero of your server resources,

One feature I really like is what Cloudflare used to call Zero Trust (I believe they're renamed it Access now). I've set it up so that any admin login is behind CF's 2FA. So the challenge to get into the admin panel, or the install directory, requires either a valid email (to which we can send the single-use login code) or a whitelisted IP address (for the staffers who typically use the same IPs). And these attempts are made on CF's servers without ever touching our own. I've also locked down any remaining WordPress installations on any servers I manage--I don't have to worry about login attempts anymore as for most of them, I am the only one with a login.

There are indeed disadvantages to CF, but some of the advantages, like this one, keep a lot of the bad actors completely away from our server now.
And I agree on letting CF take the hit on their resources to keep bots off of our servers. We're not running on large budgets, so if we can offload some of the CPU cycles to CF and let them handle the bots, rather than bogging down our own CPU and memory, that's a big plus for us also.

If CF removed their free tier and made their paid tiers too expensive, then I would consider Anubis and some of my own filtering.

 

[BrettC]

CloudFlare managed challenge on all guests. I'm interested in Anubis as a DIY solution, but for now happy with CloudFlare free plan.

Anubis is legitimately a scorched earth last-ditch option next to Cloudflare's WAF. If you want "ease of use/setup", go with Cloudflare. If you want a bit of a learning curve, and a more "i made (compiled) this, this is mine" touch, go with Anubis.

The benefit of Anubis over using Cloudflare is that all of your "stuff" is hosted on hardware that you have, or mostly have, control over. Thus, you aren't reliant on a third party's hardware potential, uptime 'guarantee', and if any connectivity/routing issues that may occur. At the same time, it's a doubled edged sword...

The drawback to using Anubis is that it does not natively support multihoming/load balancing without some hefty configuration settings (Redis/or some keyval system). Another drawback is that Anubis can become a bottleneck under extreme load, and in some instances (when I tested it with Apache JMeter & ApacheBench ab) would cripple under extreme CPU i/o. One instance of extreme-load testing did result in some sort of memory leak which resulted in the service crashing out and systemd kicking it back on.

I do like Anubis though, as it is fairly customizable and pairs nicely with other addons/modules within Nginx. Too bad Anubis wasn't a nginx module... that'd be amazing.

 

[Anthony Parsons]

Click the "Unlimited Plans" link, it's small but there on the same page, i'm on the "professional" plan.

Argh, thanks. Missed that. The next plan up from that is just silly... same specs, dedicated IP.

Have you looked at Contabo? https://contabo.com/en/vps/ pretty cheap VPS plans with more than enough specs to run your forum size. The only thing you have to do is offload your email if using domain email, to something else, otherwise if already using a free email provider, no issue. Now you are likely to get crap support, non-existent, whilst they say 4 cores, selling crowded servers is likely their thing, so you will get mixed results on your availability, no different than most cheap shared hosting, but the cores and RAM are more than needed for your small forum.

Install Centminmod and run NGINX with the latest MariaDB, let it run for a week and tune the hell out of it, and you won't have to touch the server again. As your site grows and maybe you want to use ES for search, those small plans have what you need for that, being RAM. NGINX is simple and fast. You lock it down to CF IP's only and that takes care of the majority of your issues, all to be managed via CF going forward.

The caveat, not as simple as Cpanel, but cheap and you get more specs, just need the initial setup, then there is no need to worry about the server much further.

Just giving you options, is all.

 

[BrettC]

Argh, thanks. Missed that. The next plan up from that is just silly... same specs, dedicated IP.

Have you looked at Contabo? https://contabo.com/en/vps/ pretty cheap VPS plans with more than enough specs to run your forum size. The only thing you have to do is offload your email if using domain email, to something else, otherwise if already using a free email provider, no issue. Now you are likely to get crap support, non-existent, whilst they say 4 cores, selling crowded servers is likely their thing, so you will get mixed results on your availability, no different than most cheap shared hosting, but the cores and RAM are more than needed for your small forum.

Install Centminmod and run NGINX with the latest MariaDB, let it run for a week and tune the hell out of it, and you won't have to touch the server again. As your site grows and maybe you want to use ES for search, those small plans have what you need for that, being RAM. NGINX is simple and fast. You lock it down to CF IP's only and that takes care of the majority of your issues, all to be managed via CF going forward.

The caveat, not as simple as Cpanel, but cheap and you get more specs, just need the initial setup, then there is no need to worry about the server much further.

Just giving you options, is all.

I would be wary of Contabo. I've had numerous attacks (DNS flooding/poisoning/reflection, HTTP probing/exploiting, Email spam/probing, port flooding, and DDoS based attacks) originate from their ASN (AS51167) over the past decade - and it still continues to this day. They show minimal to no desire to proactively filter / stop the malicious activity.

I would recommend Linode or Digital Ocean over Contabo. Contabo is excellent for their price-to-perf versus Linode and DO, but that $/perf comes at the expense that they have toxic IP ranges, and many providers have their whole ASN marked as toxic or are harshly filtering them.

 

[Wildcat Media]

Here's a question.

My first rule in my five CF custom rules includes an instruction to whitelist the xf_user cookie. Meaning, if a user stays logged in, they should never see a challenge. Or at least, that's what I want it to do.

(http.cookie contains "xf_user")

Action here is "Skip."

Yet I still have users saying they get a CF challenge 30 minutes later.

Is this the right way to do it?

 

[z3r010]

If it's a UAM challenge then from what I could gather it fires way before the security rules.

Position	Feature / Phase	Notes
1–2	Basic DDoS (L3/L4 + early L7)	Packet-level and initial HTTP DDoS protection
Early	Under Attack Mode (UAM)	This is where UAM fires — blanket JS challenge on all requests
3–6	Configuration Rules, URL Rewrites, Origin Rules	Early transformations
7	ddos_l7 (HTTP DDoS Protection)	More advanced L7 DDoS rules
8+	Custom Rules (Firewall)	Your custom WAF rules
9+	Rate Limiting	—
10+	WAF Managed Rules	—
11+	Super Bot Fight Mode	—
Later	Workers, Cache, Origin	—

The only way I could find around it was to give members an invisible turnstile with pre-clearance on login, allowing them to bypass further challenges.

 

[DarkGizmo]

Argh, thanks. Missed that. The next plan up from that is just silly... same specs, dedicated IP.

Have you looked at Contabo? https://contabo.com/en/vps/ pretty cheap VPS plans with more than enough specs to run your forum size. The only thing you have to do is offload your email if using domain email, to something else, otherwise if already using a free email provider, no issue. Now you are likely to get crap support, non-existent, whilst they say 4 cores, selling crowded servers is likely their thing, so you will get mixed results on your availability, no different than most cheap shared hosting, but the cores and RAM are more than needed for your small forum.

Install Centminmod and run NGINX with the latest MariaDB, let it run for a week and tune the hell out of it, and you won't have to touch the server again. As your site grows and maybe you want to use ES for search, those small plans have what you need for that, being RAM. NGINX is simple and fast. You lock it down to CF IP's only and that takes care of the majority of your issues, all to be managed via CF going forward.

The caveat, not as simple as Cpanel, but cheap and you get more specs, just need the initial setup, then there is no need to worry about the server much further.

Just giving you options, is all.

Yeah, i'd rather have a provider do all that for me tbh, if possible.

I'm still looking around and weighing my options. I have used digital ocean in the past for a game server, but that didn't need anything like cPanel.

 

[smallwheels]

digital ocean

The ASN of Digital Ocean is easily within the top 20 ASNs worldwide where continuous scraping attempts come from and there is absolutely no chance recognizable. So obviously they don't care and are thus part of the problem. Why would you give your money to a company that has part in creating the problem that makes you search for a new provider?

 

[Wildcat Media]

@z3r010 Thanks for the info--this is helping.

If it's a UAM challenge

CF custom rules

One common trend we've seen so far is that all the problems are members using their iphones on safari. Nobody else has a problem.

UAM is too drastic to keep on all the time. We only did it when we were so overrun with bots a year or two ago that it brought the server down.

As far as 3-6/Configuration Rules, etc., I would have to recheck those but even there, I can't see anything really wrong. Not unless Bot Fight Mode has gotten out of hand (no idea where that falls in this list). These members are getting a challenge every 30 minutes. That is not something I have set, that I know of.

Where would Turnstile fall in this list? Does it precede other steps here?

I imagine I'd have to figure out how to (or pay someone to) add Turnstile to the login template...?

 

[z3r010]

I used AI to add it into my UAM automation script after login (not sure it's firing as it should on my sites atm), however, I'm thinking about this now and I do think UAM fires after Configuration rules as there is an option there to turn off UAM that could be set by cookie.




If members are getting challenged every 30 minutes, change the challenge time in your secuity settings, I have mine at a week.

 

[Anthony Parsons]

The ASN of Digital Ocean is easily within the top 20 ASNs worldwide where continuous scraping attempts come from and there is absolutely no chance recognizable. So obviously they don't care and are thus part of the problem. Why would you give your money to a company that has part in creating the problem that makes you search for a new provider?

A bit unfair. Whilst I agree with the above comments about Contabo, if you are talking price, then that is the risk that comes with cheap! But whether DigitalOcean or Linode, both get used by spammers to do nasty things, along with just about every other hosting provider. The difference to my knowledge, is cheap Contabo may give you an IP used recently and blocked on spam watch lists globally, where DO and others, isolate those out until they drop off such lists.

These members are getting a challenge every 30 minutes.

You can change that.

• Log into dash.cloudflare.com
• Select your domain
• Go to Security → Settings
• Scroll down to find "Challenge Passage"
• Click the edit/pencil icon and set your preferred duration

Edit: sorry, missed the above by Z3.

 

[Wildcat Media]

I have the challenge timeout set to one week. So something else is going on.