Crazy amount of guests

[zeeb0t]

This thread has become a little too heated for my own taste, so I am not going to wade into the broader debate. Since Bot Guard has been mentioned, I just wanted to clarify a few points.

• Privacy / GDPR
  
  An EU/GDPR-friendlier operating mode is already on the backlog and scoped.
  
  I am being careful with the wording there because “compliance” is ultimately not just a product claim; it also depends on how a site operator configures the add-on, what notices they provide, what lawful basis they rely on, and so on.
  
  From the Bot Guard side, the work is understood. The trade-off is that reducing or removing longer-lived browser signals/fingerprinting means Bot Guard has less passive confidence to work with. That is entirely possible, but it would likely make the product a little more “twitchy” in terms of who needs to be challenged.
  
  That said, Bot Guard is not reliant on a single signal. It can still make useful decisions with a reduced signal set, shorter-lived identifiers, and more conservative defaults. So, this will still work just fine, really.
  
  
• Performance
  
  Performance has been a major focus, particularly recently.
  
  The newer light-DB mode was specifically introduced to move most of the decision path away from the database hot path. That does not make any origin-side add-on magically free, and I would not claim that it does. But it materially changes the cost profile.
  
  Customers who previously reported issues on very large forums have since reported stable operation while still achieving the high-precision bot removal Bot Guard was designed for.
  
  
• Origin-side protection
  
  I agree that the ideal place to shed abusive traffic is before PHP, before MySQL, and preferably before origin. That is not controversial.
  
  In fact, my own standalone Bot Guard product is designed to run outside of XenForo entirely, in front of the application. XF Bot Guard is a XenForo port of that work.
  
  But “ideal” and “available to a given forum owner” are not always the same thing.
  
  Many self-hosted XenForo admins do not have the budget, hosting control, Cloudflare tier, server expertise, or stack flexibility to implement the perfect layered setup. For them, origin-side protection may be the only practical layer they can add.
  
  So I do not see Bot Guard as a replacement for proper infrastructure, CDN rules, firewalling, caching, or rate limiting. I am very, very clear about this in the add-on/resource overview and documentation.
  
  I do see it as another defensive layer for XenForo admins who need an option that works within the reality of their setup, which is a point often missed by some.
  
  Is origin-side protection architecturally ideal? No.
  
  Can it still be a necessary and effective option for many, perhaps, even most forums? Yes.

That is really all I wanted to clarify. I am not looking to turn this into a wider argument.

Thanks

 

[Anthony Parsons]

And... this is why XF staff don't participate in stuff like this, one aggressive person who wants XF to solve all their problems for them and argues their point based on their feelings, regardless what is said, the only suitable outcome is XF solving your internet bad bot problems for you.

XF should do their thing, and developers such as @zeeb0t and @Osman have the opportunity to make great add-ons to solve a majority of the issue for those who want it solved. This is how software should work, and it does here, just like Wordpress and all other CMS / Forum builds, dev makes software, a third party market exists for devs to develop and make some $$$ at their peril.

 

[zeeb0t]

And... this is why XF staff don't participate in stuff like this, one aggressive person who wants XF to solve all their problems for them and argues their point based on their feelings, regardless what is said, the only suitable outcome is XF solving your internet bad bot problems for you.

XF should do their thing, and developers such as @zeeb0t and @Osman have the opportunity to make great add-ons to solve a majority of the issue for those who want it solved. This is how software should work, and it does here, just like Wordpress and all other CMS / Forum builds, dev makes software, a third party market exists for devs to develop and make some $$$ at their peril.

Agreed.

I am genuinely happy for each of us to play the role we are playing here.

I did not port Bot Guard to XenForo as some sort of criticism of XenForo, or because I expected the core developers to solve everything. I built it because I already had a private bot-protection product of my own, I like XenForo, and I thought a XenForo-specific version could be useful to forum owners dealing with this problem in the real world.

I also understand this problem from both sides of the fence. I have spent a long time around automation, scraping, browser behaviour, abuse patterns and the ways bots try to avoid detection. That experience is a large part of why Bot Guard is effective.

That is really the role I see add-ons playing: targeted solutions for site owners who want or need something beyond the core product.

So yes, XenForo should keep doing XenForo, and third-party developers can keep building tools around it where there is a need.

On the “make some $$$” part, I may need to revisit that master plan, because at a price point of $0 I am not exactly retiring on Bot Guard revenue.

But I enjoy building it, I like contributing something useful, and I really do like XenForo as a product. That is enough for now.

 

[ES Dev Team]

I apologize if i increased the heat level substantially. Not meant. I can be blunt to a fault.

I appreciate every single person in this discussion and hope the tit for tat on whose system is better at X/Y results in more group awareness and much better solutions for the wider internet.

For me it's been the best thread on the internet for a while because i'm intensely interested in solving the problem well.

For the record, I don't need or expect my problems solved, i'm worried about everyone else not getting theirs solved.

 

[zeeb0t]

For me it's been the best thread on the internet for a while because i'm intensely interested in solving the problem well.

Same, honestly.

I think this is one of the most important problems facing the public web right now, especially for communities that are still trying to provide a genuinely human experience.

Whether someone cares about server load, content scraping, AI ingestion, fake browsers, privacy, SEO, member trust, or just keeping their forum usable, it all sits under the same broader issue: the web is becoming increasingly automated, and a lot of site owners are being forced to deal with that reality without perfect tools, perfect infrastructure, or perfect budgets.

That is why I think threads like this matter, even when they get a bit spicy. There are different layers to the problem, and different people here are looking at it from different angles: CDN, server, application, behavioural signals, caching, crawler verification, privacy, content protection, and so on.

No single layer solves everything. But the more we share what works, what does not, and where the trade-offs are, the better the wider ecosystem gets.

And yes, I say that as someone who uses AI heavily in development too. I have been writing code for 20+ years, and my hands are quite happy to delegate some of the typing these days. But I still care a lot about preserving spaces on the web where actual humans can talk, search, read, argue, learn and build community without being drowned out or harvested by automated traffic.

So for all the heat in this thread, I do think the topic itself is worth taking seriously.

 

[ES Dev Team]

It's super serious. We don't have the public radio band of the internet if we don't have solutions for the problem that fit different needs.

I think sheer will will ensure your system becomes the top one for Xenforo, if it isn't already.

 

[Anthony Parsons]

i'm worried about everyone else not getting theirs solved.

It is a pickle of a problem. I think this thread contains some amazing information that the majority of people can solve the majority of their issue with.

 

[smallwheels]

And... this is why XF staff don't participate in stuff like this, one aggressive person who wants XF to solve all their problems for them and argues their point based on their feelings, regardless what is said, the only suitable outcome is XF solving your internet bad bot problems for you.

I assume you are talking about me - but you could not be more wrong. As I wrote more than once: I don't "expect" XF to solve "all my problems" or to implement a anti-bot solution. I think it might be a good idea if they would deliver best practices to their customers and possibly a good idea and opportunity to implement behavioral sensors into the app that could then be used for blocking. This is neither "expecting" nor "based on my feelings".

This whole thread was started nine months ago by an XF cloud customer whose small forum was run over by thousands of bots. Obviously, on XF Cloud the own possibilities for server configurations are limited more or less in the same way as on shared hosting (in fact even more due to lack to configure .htaccess yourself). The value proposition for XF Cloud is:

Bypass your IT department and have your community up and running in minutes (...) Allow our experts to manage server deployment, performance optimization, software installation, upgrades and maintenance while you focus your time and efforts on growing your community.

So at least an innocent customer would probably expect not to have to deal with the scraping bot issue personally or to be given easy to handle advice at least and basically it says: You don't need to know anything about IT to run a forum on XF Cloud.

We have seen the bots here on the forum outnumber the 100.000 regularly for months - tolerated by XF and w/o any measures agains them, as it seemed - while at the same time in this thread a lot of forum owners developed and tested strategies against bots and exchanged their experiences.

Then came the post of @Chris D in this now locked thread:

I have personally been working, or at least on call, 24/7/365 for over 12-18 months constantly having to deal with the repercussions from the type of issues you see talked about here:

We are in the process of implementing measures to prevent this kind of issue

Read what I said again. Do you think we’ve not been doing anything about it to this point?

along with measurements on this forum here that led to a visible dramatic lowering of the number of bots. Which led to the question

Will this be Cloud exclusive or will other customers be able to profit from that as well?

which was not answered.

So mentioning in this thread that XF has something in the bushes seemed pretty relevant for this thread. Due to the lack of transparency questions arose - it might be possible, that whatever it is, might even become part of 2.4 or it might be a configuration for Cloud only. Nobody knew. As I am in the process of preparing moving my forum to a different technical infrastructure somewhat relevant to me as I would not like to spend the effort building up a homegrown bot defense system if in not too distant time there might be something coming from XF that would render this effort unnecessary.

After some teeth pulling it turned out here in the thread that it seems to be "just" a configuration for XF Cloud, possibly based on Cloudflare:

That's what we've proved just by adding a couple of rules to Cloudflare?

Fine for me - so I know now, that I have to build my own setup. The more given these statements:

XenForo isn't going to help you protect your content.

privacy isn't really something that is relevant to this discussion

I'm not aiming to get the guest counts to zero, obviously. Yes, some bots might come through, but that doesn't matter.

I don't expect XF do solve the bot problem. I would like them to support their customers with solving it. And I expect a bit of transparency about what can be expected from them in not too distant future (or at all) to be able to make useful decisions where I invest my time and effort.
Independent from that I think XF should offer best practices and guidelines and they seem to think that, too. Let's hope this will happen:

We can continue improving XenForo itself. We can optimise hot paths, improve cacheability, avoid unnecessary work, and document sensible approaches.

There is probably still some heavy work along that path which, if we can optimise and make it easier/clearer/document approaches for serving at the edge, or similar, might be worthwhile.

The "behavioral API" is nothing but an idea that I consider to be interesting and possibly helpful - the more with future developments of bots - but nothing I'd expect. They don't like the idea and that's about it. Clearly nothing I'd expect or demand.

regarding aggression:

if the expectation is that XenForo should compensate for every self-hosted customer’s lack of server/CDN/firewall expertise, that is not realistic.

privacy isn't really something that is relevant to this discussion. It might be for you, but that's just another example of the need for you to implement your own solutions and to stop expecting the software to do it for you.

Your goal matters to me (and most people) a lot less than you probably expect.

If you can't do that because of constraints of your hosting, that's not an application problem, it's a you problem.

Yeah sorry for not reading nearly 1,000 posts, I don't have the time for that, unfortunately.

and from the original thread:

You had to figure out for yourself how to manage your own server? Well, yes.

Thanks for the feedback but absolutely not relevant to this case in the slightest.

Regarding the "you"-problem: I've been building up and maintaining networks, unix systems and firewalls with focus on IT security as a profession since the late nineties (and switched away from it quite a while ago to something completely different). Just because I run a small hobby forum on a shared hosting account that is sufficiently powerful for the size and purpose of the forum does not mean I would be an incompetent idiot. With the evolvement of the bots the options on shared hosting are now too limited and so I am migrating to something dedicated now, not being too keen on it as it adds another layer of work and responsibility that I'd rather have avoided - b/c I know it from experience, not b/c I would be too incompetent do deal with it.

I apologize if i increased the heat level substantially. Not meant. I can be blunt to a fault.

For the record, I don't need or expect my problems solved, i'm worried about everyone else not getting theirs solved.

Count me in on these two.

For me it's been the best thread on the internet for a while because i'm intensely interested in solving the problem well.

I don't know if it is the best thread on the internet but surely one of the most helpful threads in this forum currently. So let's get back to continue the constructive discussion rather than useless fights and insults.

 

[eva2000]

Thought I'd share an updated summary of this thread at https://botwars.centminmod.com/ for 47 pages and 928 posts Saves @Chris D and others having to read the entire thread

Enjoy!

 

[BrettC]

Thought I'd share an updated summary of this thread at https://botwars.centminmod.com/ for 47 pages and 928 posts Saves @Chris D and others having to read the entire thread

Enjoy!

View attachment 341067

"Bot War Command Center" - go on, you're got my interest now. Seems I am in the "fight it at the server" camp.

I'm more or less in both 'fight it at the edge' and the 'fight it at the server'. Edge to capture and discard most of the useless traffic, at the server to further filter the ones that got past. I'll always be in the realm of "use every tool that you have at your disposal, that you can fight with, at the enemy".

TLDR: There is no one-single-linear-way to stop malicious traffic. A properly configured multi-stage setup is required in this day and age.

 

[smallwheels]

An update from the laboratory: I did find a nice peak in the proxycheck dashboard this morning:



As it happened at 1 o'clock in the morning it was clear, that this is nor regular traffic and as it ist pretty well isolated and still small enough a good opportunity to have a closer look. The more, as it is a valid assumption that such a steep and isolated peak is probably caused by one single player and it could be interesting to see, what sources he uses.

As one can see, again proxycheck.io failed (again) to identify short to 2/3 of the traffic as bad (given the time of the day in combination with the amount one can be pretty sure that probably almost 100% of it is bad).

What proxycheck identified came to almost 100% from the US...


... and the source ASNs were all "old friends" - or rather: long known sources for enduring bad traffic, all datacenter based:

AS46261 QuickPacket, LLC ++
AS203020 HostRoyale Technologies Pvt Ltd +++
AS14061 DigitalOcean, LLC +++++++++++
AS396507 Emerald Onion (TOR) ++
AS207083 HostSlim B.V. (TOR)
AS20473 The Constant Company, LLC ++++++
AS207990 HostRoyale Technologies Pvt Ltd ++++++++++++++
AS13213 UK-2 Limited ++
AS10753 Level 3 Communications, Inc.
AS53356 Free Range Cloud Hosting Inc.
AS46475 Limestone Networks, Inc.

As usual, the constantly annoying company battled massively with the herd of degenerated royals in the huge digital ocean for the most part, while in-between some slim free range chickens picked on lime stones quickly on the search for onions (that were hidden on the 3rd level, which does only happen rarely).

So far, so boring. The more interesting part is: What did proxycheck miss - and did it get through? IP Thread Monitor sorts out IPs from blocked countries on its own (based on the local MaxMind DB) and sends the rest to proxycheck. But it does not only rely on the rating of proxycheck but also blocks the manually configured ASNs - which are a lot. So chances are that the wave was bigger than what can be seen on proxycheck and also, that not 2/3s went through but way less.

The logs said, within 2 Minutes there were 142 IP requesting access (not many more than proxycheck listed, so only 4 blocked b/c of country blocks this time) - and finally only six were let through. Which means: In addition to the 55 proxycheck attested a bad reputation another 81 were blocked by the rules within Threat Monitor and six were let pass.

All of those were residential proxies from Cable/DSL providers in the US, all did only a single request: 64.52.31.22 (AS33659 - Comcast), 143.223.148.99 (AS31942 - HyperFiber), 207.179.231.230 (AS11062 - Metro Communications Company), 206.214.134.183 (AS29866 - OPTICALTEL), 68.70.8.56 (AS14928 - PenTeleData Inc.) and 199.66.171.40 (AS13824 - AireBeam).

Those six ran then into Bot Guard and judging from Bot Guard's raw event log they probably ended up stuck there. Sounds like a success story. And a nice proof that a multi-step approach does make a lot of sense. The more, if the simple tricks like country, ASN and reputation blocking are done as far outmost as possible and only few reach the more demanding and more expensive ones, that then deal with them hopefully successfully.

I fed the whole lot into Analysis and got a wild mixture of countries, ASNs and connection types:

IP,Land,ASN,ASN_Eigentuemer
177.232.89.150,Mexiko,AS8151,Uninet S.A. de C.V.
190.54.230.58,Chile,AS22047,VTR BANDA ANCHA S.A.
168.158.139.26,USA,AS30607,WANSecurity
168.158.15.215,USA,AS30607,WANSecurity
64.52.31.22,USA,AS30607,WANSecurity
104.59.253.246,USA,AS701,Verizon Business
109.203.191.128,Deutschland,AS24940,Hetzner Online GmbH
97.251.255.14,USA,AS21928,T-Mobile USA
98.192.239.224,USA,AS7922,Comcast Cable
70.106.198.245,USA,AS21928,T-Mobile USA
137.184.225.133,USA,AS14061,DigitalOcean
209.38.68.62,USA,AS701,Verizon Business
134.199.210.10,Deutschland,AS3209,Vodafone Deutschland
185.251.46.15,Niederlande,AS208601,NForce Entertainment
203.109.63.46,Neuseeland,AS4771,Spark New Zealand
64.23.245.193,USA,AS30607,WANSecurity
178.171.126.41,Russland,AS197695,IHC.RU
104.200.68.120,USA,AS14061,DigitalOcean
137.184.5.191,USA,AS14061,DigitalOcean
100.34.32.15,USA,AS21928,T-Mobile USA
168.158.209.88,USA,AS30607,WANSecurity
152.39.151.84,USA,AS30607,WANSecurity
168.158.17.240,USA,AS30607,WANSecurity
172.117.216.126,USA,AS21928,T-Mobile USA
97.176.95.85,USA,AS21928,T-Mobile USA
174.165.210.88,USA,AS21928,T-Mobile USA
67.183.229.108,USA,AS7922,Comcast Cable
76.100.64.123,USA,AS7922,Comcast Cable
91.108.216.134,Russland,AS62014,PJSC MegaFon
75.153.64.222,USA,AS21928,T-Mobile USA
41.97.25.7,Kenia,AS36914,Liquid Telecommunications
74.110.231.20,USA,AS701,Verizon Business
176.100.139.11,Russland,AS197695,IHC.RU
143.244.179.128,USA,AS14061,DigitalOcean
137.184.9.30,USA,AS14061,DigitalOcean
213.188.90.100,Niederlande,AS43350,NForce Entertainment
93.180.199.117,Niederlande,AS43350,NForce Entertainment
180.75.240.21,Japan,AS17676,SoftBank Corp.
172.58.180.206,USA,AS21928,T-Mobile USA
178.171.48.74,Russland,AS197695,IHC.RU
136.227.166.68,USA,AS21928,T-Mobile USA
67.198.60.152,USA,AS20001,Charter Communications
158.46.201.177,Niederlande,AS208601,NForce Entertainment
98.198.148.63,USA,AS7922,Comcast Cable
76.253.69.56,USA,AS21928,T-Mobile USA
84.39.231.180,Spanien,AS6739,Vodafone España
73.207.176.201,USA,AS7922,Comcast Cable
143.223.148.99,USA,AS14061,DigitalOcean
66.135.15.49,USA,AS701,Verizon Business
92.114.61.124,Rumänien,AS39743,Voxility
180.149.0.144,Indien,AS45184,Net4India
134.199.94.136,Deutschland,AS3209,Vodafone Deutschland
207.179.231.230,USA,AS701,Verizon Business
153.66.209.194,Japan,AS17676,SoftBank Corp.
35.146.17.226,USA,AS16509,Amazon AWS
185.89.108.33,Niederlande,AS43350,NForce Entertainment
168.158.25.64,USA,AS30607,WANSecurity
75.206.70.224,USA,AS21928,T-Mobile USA
178.171.11.80,Russland,AS197695,IHC.RU
71.30.247.159,USA,AS21928,T-Mobile USA
65.186.231.51,USA,AS20001,Charter Communications
75.70.170.67,USA,AS21928,T-Mobile USA
74.67.5.206,USA,AS701,Verizon Business
31.204.10.213,Niederlande,AS43350,NForce Entertainment
185.89.104.194,Niederlande,AS43350,NForce Entertainment
158.46.210.67,Niederlande,AS208601,NForce Entertainment
162.43.239.77,USA,AS30607,WANSecurity
23.129.64.217,Niederlande,AS43350,NForce Entertainment
172.59.211.202,USA,AS21928,T-Mobile USA
174.101.120.156,USA,AS21928,T-Mobile USA
76.46.20.5,USA,AS21928,T-Mobile USA
178.171.124.135,Russland,AS197695,IHC.RU
35.145.80.99,USA,AS16509,Amazon AWS
57.135.225.189,USA,AS21928,T-Mobile USA
93.180.238.29,Niederlande,AS43350,NForce Entertainment
31.204.10.18,Niederlande,AS43350,NForce Entertainment
206.232.11.254,USA,AS701,Verizon Business
137.184.56.39,USA,AS14061,DigitalOcean
74.141.62.231,USA,AS21928,T-Mobile USA
185.240.255.133,Niederlande,AS208601,NForce Entertainment
168.158.233.143,USA,AS30607,WANSecurity
50.4.129.161,USA,AS21928,T-Mobile USA
154.29.13.78,USA,AS30607,WANSecurity
108.70.196.99,USA,AS20001,Charter Communications
178.171.8.21,Russland,AS197695,IHC.RU
95.135.22.253,Russland,AS197695,IHC.RU
185.246.175.221,Niederlande,AS208601,NForce Entertainment
198.199.69.93,USA,AS14061,DigitalOcean
188.212.137.109,Rumänien,AS39743,Voxility
194.135.133.234,Russland,AS197695,IHC.RU
154.36.70.242,USA,AS30607,WANSecurity
91.108.204.178,Russland,AS62014,PJSC MegaFon
178.171.34.102,Russland,AS197695,IHC.RU
67.1.118.232,USA,AS20001,Charter Communications
184.52.41.170,USA,AS21928,T-Mobile USA
97.71.76.58,USA,AS21928,T-Mobile USA
98.54.184.158,USA,AS7922,Comcast Cable
68.226.90.115,USA,AS20001,Charter Communications
172.58.112.234,USA,AS21928,T-Mobile USA
93.180.236.174,Niederlande,AS43350,NForce Entertainment
104.207.134.125,USA,AS30607,WANSecurity
161.123.99.161,USA,AS30607,WANSecurity
119.13.219.67,China,AS4134,China Telecom
91.108.217.246,Russland,AS62014,PJSC MegaFon
95.135.9.197,Russland,AS197695,IHC.RU
206.214.134.183,USA,AS701,Verizon Business
98.183.240.41,USA,AS21928,T-Mobile USA
216.194.85.104,USA,AS20001,Charter Communications
93.180.233.2,Niederlande,AS43350,NForce Entertainment
216.158.212.13,USA,AS30607,WANSecurity
185.223.56.96,Niederlande,AS208601,NForce Entertainment
138.68.41.206,USA,AS14061,DigitalOcean
167.99.105.200,USA,AS14061,DigitalOcean
119.12.185.104,China,AS4837,China Unicom
151.145.164.154,USA,AS21928,T-Mobile USA
167.71.114.86,USA,AS14061,DigitalOcean
104.248.184.205,USA,AS14061,DigitalOcean
185.89.104.104,Niederlande,AS43350,NForce Entertainment
68.70.8.56,USA,AS20001,Charter Communications
99.9.188.95,USA,AS21928,T-Mobile USA
178.171.7.160,Russland,AS197695,IHC.RU
172.59.194.17,USA,AS21928,T-Mobile USA
108.88.87.149,USA,AS20001,Charter Communications
161.123.183.108,USA,AS30607,WANSecurity
172.59.200.224,USA,AS21928,T-Mobile USA
69.12.83.3,USA,AS30607,WANSecurity
134.199.94.155,Deutschland,AS3209,Vodafone Deutschland
23.129.64.184,Niederlande,AS43350,NForce Entertainment
172.56.125.192,USA,AS21928,T-Mobile USA
185.81.184.132,Niederlande,AS208601,NForce Entertainment
66.17.229.142,USA,AS20001,Charter Communications
91.108.222.17,Russland,AS62014,PJSC MegaFon
66.214.7.121,USA,AS701,Verizon Business
92.255.39.160,Russland,AS197695,IHC.RU
199.66.171.40,USA,AS30607,WANSecurity
50.46.153.87,USA,AS20001,Charter Communications
98.183.2.169,USA,AS21928,T-Mobile USA
161.35.184.13,USA,AS14061,DigitalOcean
154.17.137.207,USA,AS30607,WANSecurity
134.122.13.65,USA,AS14061,DigitalOcean
91.92.17.38,Türkei,AS9121,Turk Telekom
111.225.214.141,China,AS4837,China Unicom
47.162.75.2,USA,AS701,Verizon Business

Countries like Russia, China, Turkey, India, Chile and Mexico are country blocked, so deflected directly. A bunch of US based ASNs are blocked as well (some with collateral damage as this includes quite a bunch of Cable/DSL providers and T-mobile USA - sad, but not important to my forum), others were blocked by ASNs or proxycheck. The IP-ASN alignment above was done by AI, at some point it differs from what proxycheck or other services say and it is worth mentioning, that the local maxmind db sometimes does lag a little bit in comparison to proxycheck or others regarding which ASN an IP is assigned to. IP ownership seems a little bit dynamic sometimes.

Frequency according to AI:

ASN	Eigentümer	Typ	Bedeutung
AS21928	T‑Mobile USA	Mobilfunk	many end users
AS30607	WANSecurity	Proxy / Security	many repeating ips
AS14061	DigitalOcean	Cloud	Server, Bots, Tools
AS701	Verizon Business	Backbone	commercial‑/Carrier‑Traffic
AS7922	Comcast	Kabel‑Internet	end user‑Traffic
AS20001	Charter	Kabel‑Internet	end user‑Traffic
AS197695	IHC.RU	Hosting	Server, Bots, Scraper
AS43350 / AS208601	NForce	Hosting	VPN, Proxy, DDoS‑resistent

The AI stated that NForce / Voxility‑Cluster would be well known for

• DDoS‑resistent hosting
• VPN‑/Proxy‑sevices
• Traffic‑masking

and WANSecurity‑Cluster (AS30607), officially offering VPN, Proxy‑infrastructure and Security‑Services for

• automated requests
• anonymised requests
• scraping

I've now a couple more ASNs to add to the blocking list and what we see in this wave is a very diverse access pattern throughout countries, datacenters, end user connections, mobile connections and also well known bullet proof hosters. Servers as well as resident proxies. Due to the timeframe probably centrally caused by one single player and - given the amount of dirty hosts involved - it is clear, that this player lives on the dark side of the net. Who he is and where he is based is impossible to tell.

While my approach with Add on based filtering surely won't work for huge forums it clearly does work a smaller scale. And I consider it a good solution for forums on shared hosting that want to avoid Cloudflare.

 

[ES Dev Team]

What in tarnation is a NForce / Voxility‑Cluster ?

We're writing an integration into proxycheck into our general purpose system. It has lots of categories which you can make decisions on. There's also a 'time last detected' for proxies. So from a programming perspective, this allows you to adjust the strictness.

Our system will not use this to block someone, but to raise the suspicion level for that IP. By doing that, said user will need a frontend check and then a captcha, if they pass that check.

Once you do something like ban all datacenter traffic, you need a whitelist of every datacenter/bot that's allowed to talk to your system, and use proxycheck's data on that. A pain in the ass, but possible.

 

[smallwheels]

NForce / Voxility‑Cluster

The wording was chosen by the slightly dramatizing AI. Those are two owner of ASNs that popped up with relatively many IPs in comparison to the total amount:

AS39743 Voxility (RO)
AS208601 NForce Entertainment (NL)
AS43350 NForce Entertainment (NL)

 

[smallwheels]

There's also a 'time last detected' for proxies.

Only the most notorious ones will show this. Typically either TOR hosts or other very intensively misused hosts (comment spamming, login attempts etc.). I'd say that way over 90% of the scrapers are simply labeled as "VPN" w/o special annotation and w/o history. If you bit on this flag too intensively your detection rate will suffer dramatically.

 

[smallwheels]

Once you do something like ban all datacenter traffic, you need a whitelist of every datacenter/bot that's allowed to talk to your system, and use proxycheck's data on that. A pain in the ass, but possible.

That's what I do more or less. At the size of my forum not much of an issue.

 

[BrettC]

What in tarnation is a NForce / Voxility‑Cluster ?

----

NForce Entertainment is an extremely old provider, originally starting in the Netherlands. I want to say this company got its server-starts back in 2004-2005 era, sponsoring lan parties and sponsoring various groups. They also host various enterprise game company dedicated servers. So if their ASN is becoming a wall of AI-scrapers, that's really unfortunate.

 

[smallwheels]

So if their ASN is becoming a wall of AI-scrapers, that's really unfortunate.

It would be worth double checking. While I might have seen the ASN and Name of this company infrequently at proxycheck it seems, that the AI created quite a bit of chaos with the IP-list I gave it to add ASN, Owner and country to. I.e. this IP

185.81.184.132,Niederlande,AS208601,NForce Entertainment

does, according to proxycheck belong to WS-Telecom, an US based provider which has popped up quite often in the past.
This

213.188.90.100,Niederlande,AS43350,NForce Entertainment

belongs to Host Royal and this

93.180.199.117,Niederlande,AS43350,NForce Entertainment

belongs to Host Royale as well. I did not go through the list the AI built further but if the first three are all wrong chances are the rest might be as well. I was already a bit distrusting, based on some quick tests, that's why I wrote that:

The IP-ASN alignment above was done by AI, at some point it differs from what proxycheck or other services say

But it seems the quality is way worse than I thought. Sorry for that.

 

[ES Dev Team]

But it seems the quality is way worse than I thought. Sorry for that.

Gaslit by the clanker, huh?
Generally if i need to do some analysis or data transfer like that, i'll have AI write the script, and triple check it.
Otherwise i shouldn't expect a good result, even with a big model.

 

[Anthony Parsons]

My heuristics analysis tool provided an interesting CF rule to deal with residential proxies:

Rule 3 · Challenge the rotating-proxy scraper networks​

MANAGED CHALLENGE would have matched ~9,045,287 requests in your data
Your biggest problem: scraper networks using millions of home IP addresses, 1–2 requests each, disguised as normal browsers. Their weakness: they claim OUTDATED browser versions. Your real visitors run current browsers (Chrome 147–149 right now); these networks claim versions months or years old. This rule gives an invisible browser check to outdated-version visitors arriving with no referrer. Bots fail it; a real person on an old browser passes without noticing. Logged-in members are exempt entirely.

• 601 stealth campaigns using 7,485,405 IPs made 9,973,089 requests in your data.
• Versions this rule covers (all heavily used by the networks, barely used by your real visitors): Chrome 146, Firefox 135, Chrome 145, Firefox 133, Chrome 133, Chrome 131, Chrome 120, Chrome 116, Chrome 124, Chrome 107, Chrome 104, Chrome 110…
• Traffic covered: about 9,045,287 requests.

False-positive risk (low): Roughly 2,114 requests from possible real people on outdated browsers would see the automatic check (0.02% of what the rule catches). They see a brief 'checking your browser' page, not a block, and logged-in members never see it at all.

(http.user_agent contains "Chrome/146." or http.user_agent contains "Firefox/135." or http.user_agent contains "Chrome/145." or http.user_agent contains "Firefox/133." or http.user_agent contains "Chrome/133." or http.user_agent contains "Chrome/131." or http.user_agent contains "Chrome/120." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/99." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/136." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/117." or http.user_agent contains "Chrome/135." or http.user_agent contains "Firefox/115.") and http.referer eq "" and (not cf.client.bot) and (not http.cookie contains "xf_user")

 

[zeeb0t]

My heuristics analysis tool provided an interesting CF rule to deal with residential proxies:

Rule 3 · Challenge the rotating-proxy scraper networks​

MANAGED CHALLENGE would have matched ~9,045,287 requests in your data
Your biggest problem: scraper networks using millions of home IP addresses, 1–2 requests each, disguised as normal browsers. Their weakness: they claim OUTDATED browser versions. Your real visitors run current browsers (Chrome 147–149 right now); these networks claim versions months or years old. This rule gives an invisible browser check to outdated-version visitors arriving with no referrer. Bots fail it; a real person on an old browser passes without noticing. Logged-in members are exempt entirely.

• 601 stealth campaigns using 7,485,405 IPs made 9,973,089 requests in your data.
• Versions this rule covers (all heavily used by the networks, barely used by your real visitors): Chrome 146, Firefox 135, Chrome 145, Firefox 133, Chrome 133, Chrome 131, Chrome 120, Chrome 116, Chrome 124, Chrome 107, Chrome 104, Chrome 110…
• Traffic covered: about 9,045,287 requests.

False-positive risk (low): Roughly 2,114 requests from possible real people on outdated browsers would see the automatic check (0.02% of what the rule catches). They see a brief 'checking your browser' page, not a block, and logged-in members never see it at all.

(http.user_agent contains "Chrome/146." or http.user_agent contains "Firefox/135." or http.user_agent contains "Chrome/145." or http.user_agent contains "Firefox/133." or http.user_agent contains "Chrome/133." or http.user_agent contains "Chrome/131." or http.user_agent contains "Chrome/120." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/99." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/136." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/117." or http.user_agent contains "Chrome/135." or http.user_agent contains "Firefox/115.") and http.referer eq "" and (not cf.client.bot) and (not http.cookie contains "xf_user")

The problem with this, and any user-agent based check, is that it is easily changed. Very, very easily. When I wrote bots, I actually had the docker script check what was the most installed version of Chrome, and it ran that. So, I didn't even need to change that. It may work temporarily for you - but my bet, is that over time, they will be back with a different user agent string... and probably one you cannot just block or challenge - without doing the same to most humans.