What web scraper defense techniques are you using lately?
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
What web scraper defense techniques are you using lately?
Well, since I'm on Cloud hosting and still don't want to use Cloudflare I have few options but what does help is redirecting bots in Discourage mode. It doesn't help with the bot swarms from all over the place but is quite effective for known IP ranges such as used by Bytespider and Alibaba and for the occasional scraper. The bot swarms are a nuisance but don't throttle the site and they try to go where they have no access so I never actually see them viewing anything. And they don't stay long. So yeah, not ideal, I suppose, but doing the best I can.
That is all you can do. Honestly, the XTR addon to stop all this nonsense, whilst you are doing it at the server instead of DNS, you get a small amount of load from it, but after that first block action, the load is inconsequential after that, which is better than nothing.
I am still fairly happy with the XTR add on, nonthelast (but not only) b/c of the ASN blocking feature. Has massively eased the issue even w/o Cloudflare. It took a while to tune it to its current state but I already offered @Levina to share my blocking list as a jumpstart quite a while ago.
I don't use Cloudflare plus, as I said: There is absolutely no way that a legitimate request is coming from this ASN. Apart from that: I've currently blocked roughly 500 ASNs (for the most part data centers) w/o negative effects for my forum but remarkably positive ones.
AS211273 is a new nasty one.
Nice. just four months old (21 Jan 2026). Registred in United Arab Emirates but with 7,680 Ip-Adresses, all geolocated in the US. Belonging to a company that acts as hosting provider under "hostvds.com" that has its headquarters in Hongkong and a branch office in Dubai and sells cheap VPS and storage solutions and claims to have data centers all around the world. Sounds all very honest.
China is smart, Baidu using integers to register instead of the IP, ie. 1779306888 / which is: 106.12.145.136. I find integers in my logs. I've seen it for AI bots too.
If only it could orchestrate IPTables like fail2ban does.. we could avoid instancing the PHP engine to boot xenforo, look up, then block the person.
..and use APCU for the database ( mega fast, but hard to shoehorn a baby database engine into )
If it did those things, i'd adopt it!
Well, the number of guests seem to be out of control at the moment. The swarms usually come in, stay for an hour, leave. But now we've had one since last night I think and we're now having over 30,000 of them in the house. All different IP addresses and I have nothing to battle them with.
And it's like 1500 pages of this:
And it's like 1500 pages of this:
Well, solutions have been proposed to you more than once. You decided not to make use of them. So not much to complain about but a deliberate choice I guess.
Same. I've decided not to care unless anyone has a suggestion lol
![[XTR] IP Threat Monitor](/community/data/resource_icons/10/10134.jpg?1763724168){kind=icon}
I know. I'm just a bit scared because I know so little about this. I mean, there's a reason I'm on Cloud hosting, right. And quite a bit of what was discussed in this thread went over my head. So I hesitated, things went back to normal, I waited. But now I have to really do something, because damn:
So, I will buy Osman's IP Thread Monitor. If it works on Cloud hosted sites that is. I forgot if that was mentioned at all.
That plugin is worth a try.
I'm sure you'd be able to hire someone who knows what they're doing from this forum if you are stuck.
Somewhere within this thread there is good information about setting up fail2ban, anubis, cloudflare, Threat Monitor, etc if you chose to elbow grease it.
I'm sure you'd be able to hire someone who knows what they're doing from this forum if you are stuck.
Somewhere within this thread there is good information about setting up fail2ban, anubis, cloudflare, Threat Monitor, etc if you chose to elbow grease it.
Makes no sense at all as
(which had been clarified many pages before already). This limits the options to robots.txt (useless for the problem), the IP Threat Monitor add on and Cloudflare. It is as simple as that.
There's no reason why it should not. The offer to provide you with the blocking list I am using within it still stands.
yes.
Most kind. Thanks. I'm waiting for confirmation that the IP Threat Monitor also works on Cloud. As soon as I have it I will purchase it, install it and contact you.
Do you mind if the nerds continue to cyber-squat on this thread.. 
This is actually one of the finest threads i've seen on the net about this topic
This is actually one of the finest threads i've seen on the net about this topic
XTR threat monitor works on everything, hosting has nothing to do with anything. If you have Xenforo running, its just an addon, like any other addon. You won't have access to firewalls and such, but you can use Cloudflare.
If you need help, reach out to me. I am retired, I help people for free which gives me something to do.
