Crazy amount of guests

Never really tried that route, i generally work in root-level management, not user-level management. :D
I used managed hosting myself a long time ago, and having root access was so freeing, although when I switched over it meant a few sleepless nights, worrying about getting hacked into. (A large forum I took over for was hacked at one point, but that was the early 2000s.)

I eventually got everything so buttoned up (along with killing off many sites that weren't being used) that I don't need to worry about it much. Like having two firewalls--the host has one, as do my servers, so if for some reason my rules ever fail I can fall back on theirs. At the moment I run Elasticsearch and MariaDB each on separate servers, and only access them via the local network at the host--the only IPs open to the public are 1) my own IP address and 2) the IP address of a server at a different host, so I always have a backdoor.
 
if you don't have members outside of Germany then blocking the rest of the world is one option.
For the most part a really bad idea. Except if your users never travel outside Germany.
However you have probably blocked Google and various other SEO bots, possibly some vpn users, and any payment callback providers.
Absolutely no issue to create rules for that.
In Cloudflare, you can also countries and that blocks them before they reach the server. Blocking in .htaccess means the requests still hit the server before being blocked.
Once more you are mixing up a lot of things pretty badly. In this case architecture, functionality and products. To block traffic before it reaches your server is architecture. In olden times we would have called that "having a bastion host". You don't need Cloudflare for that.
In Cloudflare, it could also be more fine tuned. Eg you can whitelist verified bots like google, bing etc and can block certain countries or just challenge others where you might have guests or members from.
This is functionality, in this case a mixture of features of a firewall and a authenticating (and potentially caching) proxy. Once more absolutely no need for Cloudflare to achieve that.
You can probably whitelist verified bots in .htaccess as well but that could make .htaccess more complicated.
Obviously every rule makes the ruleset more complicated. That is no different between .htaccess and Cloudflare. It is pretty trivial to add these rules to .htaccess and makes it in no way complicated.
It would mean setting up a free account in Cloudflare and having your dns go through there. It's not a big job.
And here we go again: If the only took you now is a hammer you recommend it als the only tool to everyone and if you only have a hammer everything looks like a nail. Cloudflare is the solution for all and everyhing. :rolleyes: The world of tools is bigger than that. No mentioning of the downsides, no mention of alternatives not even asking if Cloudlare could be an option at all, just blindly recommending a product. As said often enough in this thread: They do offer a product that fits many needs but they are by far not the only ones, they are not the only way of doing it and they do have their downsides. Ignoring all that again and again destpite, after it being mentioned countless times, is nothing but ignorance and everything but helpful.
 
My response was to @darkeye. You've made it well-known you don't like Cloudflare @smallwheels but plenty of other people do and for the reasons given. This isn't your forum and you're not the moderator - so we are all free to reply to others. Indeed most forums ask posters to keep it friendly and not personal or critical. I think undermining those suggestions doesn't help the poster asking for information, and particularly as your post didn't offer anything constructive to the poster.
 
You've made it well-known you don't like Cloudflare
I cannot "not like it" as I don't have enough experience with it to judge in such a way. I see it critical for a number of reasons that I've outlined often enough, one of them is them becoming close to a monpoly. One conseqence of that is that obviously attackers focus on outmaneuvering CF (which would be a good reason to go for a less prominent alternative) and secondly we have seen just this week the consequences of such a monopoly and what happens when a monopolist fails - in this case microsoft. Cf did fail, too in the past and it will for sure again in future.
Still it might be a good product and suit the needs of many. But people should make informed decisions. One of the big downsides of many forums is that people with half-decent knowledge give advice to people with no knowledge and this way often some (sometimes bad) products get hyped uncritically while other (sometimes better) solutions are not even mentioned. That is the nature of forums. However: In a technical world this is not desirable. When obvious and well known disadvantages of a solution are not mentioned even deliberately to hype the own solution this is simply disinformation. Statements like "your .htaccess will become complicated when you want to let pass Google" when at the same time you have not even seen a .htaccess from afar (b/c otherwise you would not make such a statement) are in the same league.
You may have best intentions and it is totally fine to say that you are happy with c/f - but please: don't pretend to a networking and firewall guru that you are not. Happy if CF works for you - but all forums that I've seen that use CF end up still having 1.000 - 2.000 guests - which for me is simply not good enough, independend from any other criticisms that I may have or not and independent from wether I "like" it or not.
You have once more promoted a solution (or rather a product) w/o being asked for a problem which wasn't described comprehensively in an environment that wasn't described either and for which no desired goal was given either. Instead of asking all these things to better understand the situation and the needs you blindly fire out a strong recommendation for your favored product. The only product you know. Hyped and uncritcally only mentioning the sunniest sides. This is no advice, this is blind hyping.
your post didn't offer anything constructive to the poster.
My post went towards you, not the poster. Even if I wanted to give him advice I've not enough information to do so. His post does not even meniton if he is on shared hosting or a server. In fact I gave him helpful advice by providing a counterweight to your advertizing. Fun is: He did not even ask for advice. He told his story and shared his .htaccessrules as a potential help for others and asked one question:

Maybe someone can suggest improvements to my .htaccess and robots.txt configurations. Any feedback or suggestions are welcome!
That's it. You did not deliver that in the slightest (and obviously did not even realize that he did not ask for advice) but instead once more blindly tried to talk him into CF w/o even asking about his goals and situation.

PS: Regarding "you hate Cloudflare and give no advice": This is probably why I wrote an overview about the most common solutions regarding bots not even two weeks ago in this very thread - including Cloudflare:

 
Earlier in this thread, Cloudflare was a fairly major recommendation from many people, not just myself. And it can hardly fail with a hard block!!! Challenges aren't foolproof but a block is a block - at the edge, before it reaches the server. Within that block it can whitelist verified bots, such as search engines, and registered members. Which could be useful if a member happens to be visiting another country.

I am aware of some other products and own another one but haven't needed it. But yes Cloudflare works well for me and keeps the bots out without restricting most real people.
 
Last edited:
Earlier in this thread, Cloudflare was a fairly major recommendation from many people, not just myself.
It still is, obviously.
And it can hardly fail with a hard block!!!
You don't need CF for that and while I agree regarding the benefits of a hard block (and use this massively) quite a lot of people here in this thread have a way different opinion regarding that.
at the edge, before it reaches the server.
Once mor you mixing things up. What exactly is "the server"? The computer? a virtual machine? Firewall? Apache/Nginx? XenForo? Which of these comes to it's limit under which conditions and why? If none of them do you don't need to filter at CF. It you can't answer the question better make your homework before creating a more complex setup. "The edge" is btw. not necessarily CF, it simply is the furthest out part of your setup, as far away from your application as possible. It may simply be the firewall on your server itself or another host close to it. We are in technical debate where words have a meaning. Mixing those meanings up by blindly repeating words causes confusion and disinformation.
The advantage of CF is that it is a CDN like other CDNs, too (so can migitate load through caching) and (as others) has filtering options, so can in sum protect against DDOS (which has become a rarity today but some bot waves seem to have similar effects). If you don't need that there is no need for CF. You can still use it to as kind of an insurance to fire it up in case a nasty huge wave hits you - however you do pay a price for that through a more complex setup.
Within that block it can whitelist verified bots, such as search engines, and registered members. Which could be useful if a member happens to be visiting another country.
Again: You don't need CF for that in the slightest.

BTW: What you completely ignored in your praise for CF: The OP is and probably hosts in Germany. Germany's biggest internet provider for end users is DTAG. DTAG has no direct peering with CF due to DTAG's arrogant peering policy. Which means: In certain situations using CF can be counterproductive if your main user base is in Germany. Pretty important aspect for the poster, isn't it?

Again: I don't say that CF would be a bad option in general or not a potential option for the poster. It just makes absolutely no sense to give flat out advice in complete ignorance or lack of knowlege of the situation, environment and goals of the one having the problems. Depending from that CF may be advisable or not.
 
Last edited:
I just checked that and it changed in September this year and they now have a partnership. But yes GDPR would apply in terms and privacy notices/cookie notices, but apparently Cloudflare deals with the agreement automatically. Storing of IP addresses can be an issue with GDPR whatever method is used.

 
I had three IP's registered today - but the Registration Spaminator caught them. They had bypassed cloudflare rules due to the route they took. All made identical requests. The spaminator details showed they would have pasted some very dodgy links (they pasted those in a registration field) - one relating to the dark web.


They were all hosting company IP's

These two were from ASN 55286 Servermania.com, Buffalo New York

198.245.67.137
38.154.145.205


This one was from ASN 216701 Servers Tech FZCO, Amsterdam

195.200.28.67

Both ASN's added to a Cloudflare challenge now - they can't solve that as they didn't run real browsers and were simple bots. Challenged instead of blocked in case of a rare VPN user on the ASN.
 
Last edited:
I had three IP's registered today - but the Registration Spaminator caught them. They had bypassed cloudflare rules due to the route they took. All made identical requests. The spaminator details showed they would have pasted some very dodgy links (they pasted those in a registration field) - one relating to the dark web.


They were all hosting company IP's

These two were from ASN 55286 Servermania.com, Buffalo New York

198.245.67.137
38.154.145.205


This one was from ASN 216701 Servers Tech FZCO, Amsterdam

195.200.28.67

Both ASN's added to a Cloudflare challenge now - they can't solve that as they didn't run real browsers and were simple bots. Challenged instead of blocked in case of a rare VPN user on the ASN.
I see no reason why we need servermania traffic hitting our server, they have been firewall blocked on our server for years-
209.127.0.0/19, 209.127.32.0/20, 209.127.48.0/24, 192.186.128.0/18, 161.115.232.0/21, 23.229.0.0/17, 23.236.128.0/17, 152.232.0.0/16, 38.154.4.0/22, 198.20.160.0/19

Also, use FireHol list https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset to block nefarious web traffic. Make sure if you use a CDN like Bunny, you have the script that imports the Firehol rules exclude the addresses from your CDN like Bunny https://api.bunny.net/system/edgeserverlist/plain because they land on the block list often.

I used Gemini to create a powershell script and it works perfectly!
 
Last edited:
I see no reason why we need servermania traffic hitting our server, they have been firewall blocked on our server for years-
209.127.0.0/19, 209.127.32.0/20, 209.127.48.0/24, 192.186.128.0/18, 161.115.232.0/21, 23.229.0.0/17, 23.236.128.0/17, 152.232.0.0/16, 38.154.4.0/22, 198.20.160.0/19

Also, use FireHol list https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset to block nefarious web traffic. Make sure if you use a CDN like Bunny, you have the script that imports the Firehol rules exclude the addresses from your CDN like Bunny https://api.bunny.net/system/edgeserverlist/plain because they land on the block list often.

I used Gemini to create a powershell script and it works perfectly!
Careful if you're using the FireHol L1 blocklist, as it's also filtering out bogons and restricted-use IP CIDRs. That can sometimes block internal services/routes (VPNs for example). It probably would be better to use the FireHol L2; https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset

Buuuuuut.... since we're on about FireHol, check out their other block lists that they list:

 
Well, not so good news... as of this week, it looks like our residential proxy ai-scraping botnet friends have pushed an update to their scrapers. The first indications a short time back seemed to be the bot runners reconfiguring and 'testing the waters'.

From what I can see, these bots no longer have any issue passing JavaScript and are solving Anubis. As a test, they were able to solve a challenge level of 7 in a minute and a half or so. Such an elevated challenge level would take many minutes to solve for a normal client. While Anubis is still technically doing its job, this is starting to become a losing battle.

Up until this week in 2026, it was simple as filtering at the Anubis level, or blocking a toxic IP range/CIDR, and in worst cases, the whole ASN's advertised IP CIDR list. Sure, I can add another layer of protection, such as Cloudflare... it just won't stop the elephant in the room: The issue of residential proxies is exceptionally vast, and is an actual problem at this point.

Edit: forgot to mention that Anubis is still having no issue blocking the badly coded bots, probers, and exploit bots. So there's that. :-)
 
Despite not directly tied to the scraper topic some additional bad news that is probably worth mentioning b/c worrying for those running their own servers: Attackers are now using AI to scan freshly related security patches to develop attacks before systems have been patched:

Hackers are now using LLMs to continuously scan software updates, spot newly fixed bugs, and build attacks before teams can apply the fix — all without human involvement. That’s according to the latest report from the Google Threat Intelligence Group.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.



1794992042896.webp


1794992447463.webp
 
Last edited:
Well, not so good news...
Can confirm - just yesterday I caught a few dozen bots trying to get into my private forum, hitting the same URL simultaneously through two different proxies in completely different parts of the world. Passed all Cloudflare filters for ASN/country, for dubious user-agent values and the captcha for not being authenticated; passed Bot Guard's filters as well.

At the speed this is relentlessly escalating, we might (hopefully) get some traction in two possible scenarios:
  • new RBLs gathering distributed intel over residential proxies and getting them blacklisted on the internet, maybe getting ISPs to react (not holding my breath on that one)
  • as this starts looking more and more like fully automated "hacking" by searching for credentials to abuse (for cryptomining or further botnet expansion) and exploiting vulnerabilities, some nations' security agencies might start to get involved, and these guys would not tread lightly.
Who had something on the note of "fighting for the survival of Internet" among their "where do you see yourself in 5 years" answers 4 years ago? :)
 
I'm still managing to stop the solvers in Cloudflare :-) Hope that continues. Which is why I'm maybe getting spambots now instead. Logs also showed a couple of dodgy scanners looking for vulnerabilities. The server blocked them, but I blocked them in Cloudflare as well, just in case.
 
Last edited:
That can sometimes block internal services/routes (VPNs for example).
The only issue I noticed is that it would block Bunny CDN sometimes. That is why the script cross checks the Bunny List- https://api.bunny.net/system/edgeserverlist/plain

As for people using a VPN, VPN is kind of like the 1990s blocking your caller ID, nobody would answer. There is no reason for someone to be on our websites using a VPN, especially for e-commerce or forum registration. Nearly always associated with something nefarious.
 
I thought that about vpn users for a while too, but these days, people are pushed vpn's all over the place for "security" - there may be a lot more people using them these days. And also I have some countries blocked, and the only way real guests could access my site from there is using a vpn. I had one member from an eastern block country that messaged to say she could only access the site using a vpn, even to log in.
 
I had one member from an eastern block country that messaged to say she could only access the site using a vpn, even to log in.
Fortunately, our community’s audience is primarily focused in the US, Canada, and UK. For me, from a security standpoint, the risk-reward ratio simply doesn't justify allowing VPNs that are being recently used for nefarious activities—especially given the rise in active threats and the revelation that PHP vulnerabilities existed for many years that were only recently discovered by A.I.. While I understand this might affect users in restricted regions, managing those security and compliance risks isn't feasible for us. We aren't willing to accommodate workarounds for users whose local laws restrict access to our website.
 
Back
Top Bottom