Crazy amount of guests

[Alvin63]

What timeframe do those logs cover?

The last 9 days

 

[Alvin63]

Just to add the way the bots are doing this - only 2 to 5 requests from an IP and never repeating the IP, means rate limiting rules won't work. They are successfully evading any form of rate limiting by using so many different IP's with a long period in between.

 

[BrettC]

Just to add the way the bots are doing this - only 2 to 5 requests from an IP and never repeating the IP, means rate limiting rules won't work. They are successfully evading any form of rate limiting by using so many different IP's with a long period in between.

Your only options at this point is strict cloudflare rules or Anubis... and that's ignoring any xenforo guest-access restrictions.

 

[smallwheels]

Just to add the way the bots are doing this - only 2 to 5 requests from an IP and never repeating the IP, means rate limiting rules won't work. They are successfully evading any form of rate limiting by using so many different IP's with a long period in between.

That is basically what this whole thread is about for 65 pages... And it is the reason why IP Threat Monitor got better blocking via ASNs, Countries and IPs shortly after release as it turned out that rate limiting is sometimes still a thing, but not too often.

 

[smallwheels]

Your only options at this point is strict cloudflare rules or Anubis

No. As I said countless times: there are many ways to Rome. Plus once more you ignore the fact that @Alvin63 is running on shared hosting - if you have a way to install Anubis there please let us know... Given his level of technical expertise I would have my doubts regarding Anubis anyway.

 

[Alvin63]

Given his level of technical expertise

I'm female Not that it's particularly relevant. The username is from our son's favourite TV programme Alvin and the Chipmunks (it's in a profile post somewhere).

 

[Alvin63]

I'm intending on doing the Cloudflare secret word header and the corresponding .htaccess command.

@badabugs1 please could you confirm the exact placement for the code within the existing Xenforo .htaccess text? As obviously I'd need to get that right. There are currently two hard L's - one Xenforo and one from the secret header code so I'd need to make sure I put it in the right place. And yes I would test the header is working first before adding the .htaccess code!

 

[BrettC]

No. As I said countless times: there are many ways to Rome. Plus once more you ignore the fact that @Alvin63 is running on shared hosting - if you have a way to install Anubis there please let us know... Given his level of technical expertise I would have my doubts regarding Anubis anyway.

If their shared hosting provider is using cPanel (which appears to be such), and said hosting provider is using Litespeed (unsure if this is active) for the web server, https://docs.litespeedtech.com/lsws/cp/cpanel/anubis/ - it's a native module and can be enabled.

Otherwise, migrating their forum from a shared provider and moving to a VPS would be an option. Many VPS hosts even offer prebuilt ecosystems with the full LEMP or LAMP feature set. With the right amount of optimizations and techniques applied, VPS hosting might actually be better for their community, but the word 'might' carries the most weight.

For whatever it is worth, Anubis can be configured to be highly selective on what is challenged, it's not an all-or-nothing WAF. For example, i could force a challenge to particular IP ranges or guest clients on all ./threads/* ./whats-new/* and so forth, but nothing else - highly configurable.

 

[Anthony Parsons]

Just to add the way the bots are doing this - only 2 to 5 requests from an IP and never repeating the IP, means rate limiting rules won't work. They are successfully evading any form of rate limiting by using so many different IP's with a long period in between.

All that's happening is you're discovering exactly what this thread has talked about the entire time. Residential Proxies.

I think you need to focus on one working solution at a time.

You either need to get your server locked to CF IP's only OR you need to use something like Threat Monitor which won't affect your server load, follow instructions, download maxmind to your server and use geo blocking directly. Then just take out CN entirely, no more issues. At the origin is not the best solution, but if its the only workable solution, its better than nothing.

If you can get the workaround, similar to authenticated origin pulls via htaccess working, that is a solid solution.

You will not beat residential proxies with IP or ASN blocking. You're better of to get a cheap VPS for $10 a month so you can manage your server, lock it successfully to CF, authenticated origin pulls and control the issue in CF WAF or similar DNS system. CF just make it free and sort of easy, to do, providing you do your research and us AI to help you along the way will make it easier as well. You still have to understand what you are doing though, otherwise AI will walk you into places you don't want to go.

 

[Levina]

You will not beat residential proxies with IP or ASN blocking.

No indeed. I started with geo-blocking and blocking ASN's. That worked for a short while until the residential proxies came in. Block paths that you can block. Put a Managed challenge on others. Don't make Search available to guests. And be very critical of what AI advises you!!

 

[Alvin63]

All that's happening is you're discovering exactly what this thread has talked about the entire time. Residential Proxies.

I think you need to focus on one working solution at a time.

You either need to get your server locked to CF IP's only OR you need to use something like Threat Monitor which won't affect your server load, follow instructions, download maxmind to your server and use geo blocking directly. Then just take out CN entirely, no more issues. At the origin is not the best solution, but if its the only workable solution, its better than nothing.

If you can get the workaround, similar to authenticated origin pulls via htaccess working, that is a solid solution.

You will not beat residential proxies with IP or ASN blocking. You're better of to get a cheap VPS for $10 a month so you can manage your server, lock it successfully to CF, authenticated origin pulls and control the issue in CF WAF or similar DNS system. CF just make is free and sort of easy, to do, providing you do your research and us AI to help you along the way will make it easier as well. You still have to understand what you are doing though, otherwise AI will walk you into places you don't want to go.

Thanks. Yes I've been focusing on the Cloudlfare rules and next the .htaccess if things don't improve. And yes I can see now it's all about residential proxies. A VPS would be better but I don't really have the technical expertise to manage one and managed options are too expensive for my small site.

 

[Alvin63]

No indeed. I started with geo-blocking and blocking ASN's. That worked for a short while until the residential proxies came in. Block paths that you can block. Put a Managed challenge on others. Don't make Search available to guests. And be very critical of what AI advises you!!

Thanks Indeed that's why I wanted someone on here to check the code! However Claude is quite good for analysis.

 

[Anthony Parsons]

A VPS would be better but I don't really have the technical expertise to manage one and managed options are too expensive for my small site.

I am retired, and I am happy to help you freely if you want to get onto a VPS. About to go on holidays for a week tomorrow... if you need help, let me know.

 

[Alvin63]

Thank you that's very kind

 

[puterfixer]

A long time ago, when I added CloudFlare in front of my domain, I did all the preparation steps, including:

• proxying DNS for the essential domain records (A, AAAA and CNAME) in CloudFlare, under DNS - Records;
• enabling edge certificates and deploying the SSL certificates on the shared hosting;
• enabling TLS 1.2 as a minimum version and activating TLS 1.3 as well;
• whitelisting CloudFlare IPs in .htaccess and denying connections from elsewhere.

All was well in my world, until one day a few weeks ago, the hosting suddenly started throwing "not allowed" on any page, as if I was bypassing CloudFlare. I checked DNS records and caches and IPs and everything on my computer, to no avail - in the end all I could do was to remove the line blocking all other sources in .htaccess and traffic would pass through again. No idea what was wrong, I suspect some update at the shared host which started filtering by client's origin IP in the header or something, which CloudFlare was passing on to the hosting so the actual visitors' IPs could be tracked (otherwise in Who's Online I would see for every visitor the IP of the CloudFlare intermediary server).

But now the trick with the secret additional HTTP header between the CloudFlare hop and the shared host, this is absolutely genius. I implemented it already and it seems to be allowing traffic reaching the hosting server through CloudFlare, and it is not affected by whatever issue was preventing the IP-based filtering to work on my shared host.

This is my .htaccess as an example, doing a few things explained in comments:

# No explicit static pages being served for various HTTP error messages
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 405 default
ErrorDocument 406 default
ErrorDocument 500 default
ErrorDocument 501 default
ErrorDocument 503 default

# Ensuring all requests go to the domain without www. in front
RewriteCond %{HTTP_HOST} !^mydomain.tld [NC]
RewriteRule ^/?(.*)$ https://mydomain.tld/$1 [L,R=301]

# No more unencrypted HTTP access, all traffic redirected to HTTPS
# This can also be enabled directly in CloudFlare for example
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

<IfModule mod_rewrite.c>
    RewriteEngine On

    # This is the header and secret key inserted by CloudFlare to certify traffic coming from CF
    RewriteCond %{HTTP:X-My-Secret-Header-Key} !^bazinga$
    RewriteRule ^ - [F,L]

    # This is to allow direct access to files only in specific locations, otherwise force probing for
    # files in random locations to go through index.php
    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -l [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteRule ^.*$ - [NC,L]
    RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L]
    RewriteRule ^.*$ index.php [NC,L]
</IfModule>

On the topic of CF filters based on IP, keep in mind that it's imperfect. The subnets of IPs can be traded among organizations any day, and the databases that maintain the relationship between subnet - AS Number - geographical location/country code are not updated in real time. It has happened before that, depending which registry I was querying, an IP was listed as belonging to one organization and country, or another organization and country. And that's how your ban for an ASN might not apply to a specific subnet because in CloudFlare's source the subnet is assigned to another ASN, however if you query another source you find a different result.

Also, it was stated before that China in particular is doing some dodgy stuff at networking layer on the internet, deliberately misusing/poisoning the BGP protocol between major data carriers to manipulate their routing tables and make some subnets as belonging to other ISPs or countries. They do that to bypass existing filters in various places that are blocking Chinese traffic; it is arguably abusive or fraudulent, but definitely effective. And if governmental actors are struggling to defend their infrastructure from chinese probing bypassing big guys' defenses, don't start pulling your hair out when you, a little fish in the Sea of Internets, can't do any better with the limited tools available for free.

Here's a better explanation of what BGP hijacking is - feel free to skim through it if it gets too technical:

https://www.cloudflare.com/learning/security/glossary/bgp-hijacking/

There is even live monitoring of BGP routes on the internet, and the current "anomalies" of IP hijacking being done:

bgp

bgpwatch.cgtf.net

Here's a deeper technical article about this, but towards the end it showcases a few of the big real-world BGP hijacking of past years:

BGP Hijacking: What It Is, How It Works, Real Examples, and How to Prevent It | Anapaya

BGP hijacking redirects Internet traffic through false route announcements. Learn how BGP (Border Gateway Protocol) hijacking works, the different types of attacks, and how SCION eliminates routing attacks at the protocol level.

www.anapaya.net

And there are more examples. In 2016, governmental traffic between Canada and South Korea (probably including diplomatic conversations but also military inteligence data exchange between allies) was routed through Chinese infrastructure for about 6 months. In 2024, China basically hijacked the European traffic to make it pass through its own routers for about 2 hours (until it was detected and corrected), most likely logging everything to break encryption and to analyze all contents, looking for anything secret or useful that it could exploit for commercial, political or criminal gain.

There is a quiet war going on, and we don't even see it, just sometimes get glimpses of it when our CF filters don't always work. Well, this is why.

 

[Alvin63]

That is really interesting. It got me curious as to who my six user agents were behind it all and that can't be determined because all six use normal things like Mozilla, Chrome etc - because they are residential proxies. It made me think legislation is needed as regards residential proxies. I chatted with Claude about that and was surprised to hear that it is actually legal for companies in most jurisdictions, to sell lists of residential proxies. Providing they show they had consent. And that consent apparently is hidden in app terms that probably no one reads - in things like free VPN's (although maybe other software people unknowingly install. And the company or person behind my worldwide residential proxy operation could well be operating a legal business.

So I think legislation is needed for some of these companies to be more upfront about what someone is consenting to. A free VPN isn't technically free if that is the case.

 

[smallwheels]

all six use normal things like Mozilla, Chrome etc - because they are residential proxies.

Not really. The user agent can be set freely, it would be no issue to set up requests from a user agent "Alvin's hamster browser". There is absolutely no reason for a certain user agent b/c of residential proxies. They try do hide by using a "normal" agent, but often fail to do so comically. So a certain, uncommon user agent string may be an indicator for a certain actor or a certain toolchain, but nothing more.

I chatted with Claude about that and was surprised to hear that it is actually legal for companies in most jurisdictions, to sell lists of residential proxies. Providing they show they had consent. And that consent apparently is hidden in app terms that probably no one reads - in things like free VPN's (although maybe other software people unknowingly install. And the company or person behind my worldwide residential proxy operation could well be operating a legal business.

So I think legislation is needed for some of these companies to be more upfront about what someone is consenting to. A free VPN isn't technically free if that is the case.

Once more you "discover" things via you AI that have been excessively discussed insight out in this very thread for ages already. Maybe you should take the time and read the thread you are posting to, especially if you want to learn about the matter. You seem to trust AI a little bit too much and while this thing may be an exciting learning journey for you, eased up by AI, it is not so much for everyone else who has been participating in this thread vor many months. The more, as the typical AI slop sounds as confident as convincing but - as usual - it not as easy as the AI claims (or you understood it). There is probably a reason why many of these proxy sellers formally reside in countries like Cyprus or UAE. And yes, there are cross connections between them and VPN providers, some shady networks, app programmers and also crypto bros. It is a very filthy business and it seems as if many of those providers are interconnected, sharing their IP pools and possibly only different storefronts, run/owned by the same people. However: While I consider all of them filthy they still are not all the same and some are - astonishingly - more or less well known, kind of "reutable" companies. So read the thread you are posting to and you'll find out.

 

[Alvin63]

Can you quote me someone else on this thread saying "So I think legislation is needed for some of these companies to be more upfront about what someone is consenting to. A free VPN isn't technically free if that is the case." It was a conversational point on the topic. I will admit I haven't read the entire thread - which is 65 pages long - although have read a lot of it.

And maybe peoples' circumstances are all slightly different even with common issues.

 

[puterfixer]

Guys, be cool. We're in the same boat.

There are over 200 countries on this planet, there's not a single legislation which magically can resolve this. There is pressure on manufacturers of smart devices, like smart TVs for example, to validate more closely the applications that are published in their appstores and prevent the use of their devices as a potential distributed botnet, proxy etc. Even if the consumer is asked to give their consent, most of the time they have no idea what they're consenting to.

Just a fresh example, LG has been at the heart of an ongoing scandal in which they use their smart TVs to literally record audio of what people say in their homes, they scan the home network to identify what other devices do they have, and this way to feed more data into their own advertising network. The LG Ad Network's CEO, banned by other organizations after lots of dodgy stuff, is claiming that "we own the glass in the livingroom" - the "glass" (screen) which people paid thousands of pebbles to own. LG claims that the consumer has the option to refuse the terms of use which gives consent to this level of tracking of everyone in your home, but then the TV is nearly unusable and continuously nags you to accept.

The residential proxy stuff is not a standalone application; it's a component added to various apparently innocent apps, like games, which are "free". Instead of getting revenue from advertising, the developers of those games get revenue from allowing their apps to be bundled with the proxy functionality. The proxy access is then managed by some commercial entities in countries far far away, which get paid by scrapers to let them use the proxies for harvesting internet contents, and a small part of this revenue trickles down to the game developers.

Smart TVs in particular are more appealing to run such applications than a home PC; they have very capable CPUs and operating system, unlike other much more specialized "smart" appliances, and they offer no way for the consumer to monitor what's going on in there (as one could do on a PC). Unlike computers, TVs are powered 24/7 and have internet access typically in a home which does not have the advanced networking equipment to observe that the TV is transferring gigabytes of data per day even when it's in stand-by (which could as well mean just display and audio amplifier powered off, but the CPU running).

We're getting to the point where some people have been advising for years to never ever ever hook up a smart TV to the wired or wireless network in a home, and the manufacturers are learning to circumvent that - they are actively searching for open hotspots to connect to automatically or to create a wireless mesh with TVs from the neighbors which might have internet connection. It is very, very, very intrusive, and there is no one legislative body that can control that. Especially not in the States which are protecting the interests of corporations, not the consumer.

Regarding the 6 user-agent strings, that might simply be a list within the scraping tool gathering data through the proxies, or perhaps there are 6 different variants/editions of scraping tools. The answer to "why 6" doesn't help in any way; at most we can target those specifically with rules, but it's a temporary measure until they start using another one, another 6 or another 100 user agents, real or not.

 

[smallwheels]

Can you quote me someone else on this thread saying "So I think legislation is needed for some of these companies to be more upfront about what someone is consenting to. A free VPN isn't technically free if that is the case."

Probably not, b/c this is very wishful thinking. You can assume that the conditions are clear upfront typically, but buried deep in the TOS, not much different as any insurance company, car dealership or other business buries shady practices deep in their TOS and have gotten away with it for decades. For one, as governments don't act and secondly, b/c users don't read what they agree to. If this does not work with a plain national entity like a car dealership, how well would this work with businesses that are spread out through countless countries world wide (and preferably those with a very leger company legislation), typically hide the real actors behind them and change owners, actors, names and business locations frequently? No way that legislation could hinder them, the more as they don't care anyway and, if in doubt, run faster than the law. At least some of them are over the line to illegitimate businesses are very close if not over the line to organized crime, especially with connections to hacker groups, DDOS attacks, crypto-schemes, spyware and more.

As long as people do not understand that there is rarely free beer available w/o conditions those businesses will succeed. Free VPNs are probably only a fraction of the problem, things like proxies hidden in games APKs (and distributed with free games) or integrated into your Chinese TV stick from Amazon or Alibaba are probably the more common issue.

These guys i.e. ...

View attachment 335633

...claim nothing could stop them and that they'd have 110.000.000 proxies on offer. The address of their webpage resolves to .... Cloudflare. So possibly it'd be a good start if Couldflare would stop supporting the shady businesses whose practices they fight as a business.
But maybe it's even deliberate: The more of these services flood innocent websites the more customers Cloudflare will get.

I became curious and checked a couple of Sellers of Residential Proxies (the first hits on google) regarding where their own IP addresses belong to:

Name	No. of Proxies (claimed)	No. of customers (claimed)	website resolves to IP from	company location according to website	Domain registered in	domain created / updated
scrape.do	110 Mio	?	cloudflare	US (but probably Turkey)	? / redacted	2020 / 2024
bright-data.com	440 Mio+	20k+	cloudflare	Israel	godaddy / US privacy protection service	2015 / 2026
oxylabs.io	177 Mio+	15k+	cloudflare	LT	101domains / redacted	2015 / 2020
webshare.io	80 Mio+	60k+ paid / 120k active	cloudflare	US / California	namecheap / unknown privacy protectn service	2017 / 2025
proxy-seller.com	20 Mio+	?	cloudflare	Cyprus	namecheap / Cyprus	2015 / 2025
evomi.com	30 Mio+	?	Framer B.V. / AS16509 (Amazon.com, Inc.)	CH	tucows / privacy protection service	2014 / 2026
iproyal.com	32 Mio+	?	cloudflare	UAE	namers / SE privacy protection service	2018 / -
dataimpulse.com	90 Mio+	?	cloudflare	US / Florida	godaddy /US privacy protection service	2008 / 2025
proxyrack.com	12 Mio+	?	AS16509 (Amazon.com, Inc.)	Fake Office in Hongkong	namecheap / IS privacy protectn service	2012 / 2024

All of these services claim to only have ethically sourced proxies. The majority hides behind Cloudflare and two are on Amazon. Which means: Cloudflare actively supports or at least tolerates scraping companies as their customers. This should in my eyes be a bit of a warning sign.

I found a new kid on the block today in my logs: A bunch of IPs coming from "AS59651 Alex Largman", registered in Israel and partly subleased to a "Beyond Tomorrow Ltd", trying to gain access from various countries. Had not had this ASN in my logs before, so I became a bit curious. As often, a lot remains unclear but then I found a bit of gold. Upon other findings the name showed up in an article that followed the path of a DDOS attack in 2023 and discussed the involvement of companies selling proxy access in those attacks and followed the path to trace down the people behind these companies.

Alex Largman turned out to have changed his name from "Aleksey Bulgakov" and is the founder of a company named "Fast Internet LLC" - he even has a Github account under this name. There is a company with this name run by a person named Alex Largman registered in the UK and classified as a micro company. According to the article...

Proxy providers weaponized to launch denial of service attack against Rappler – Qurium Media Foundation

www.qurium.org

... most of the players behind these services seem to come from Russia (and some from the US) and have a long history when it comes to dubious and fraudulent activities on the internet. It is a pretty interesting read how they found short-lived companies around the world to gain IP-netblocks and cover their activities, how involved they are in attacks with their proxy companies and how "ethical" their sourcing really is.

Quirium regularly posts forensics about DDOS attacks and I cross read a couple of those, only to find more proxy companies to be involved - any many of the company names mentioned sounded pretty familiar. Like these here:

Proxy provider connected to state sanctioned research center linked to attack against investigative media – Qurium Media Foundation

www.qurium.org

I.e. in one article 3xK Tech GmbH showed, a company from Germany that I mentioned way further up this thread as it seemed a bit of a strange company - until today they are a regular guest in my log files for blocked access attempts:

Yet another proxy provider behind the iStories DoS attacks – Qurium Media Foundation

www.qurium.org

So the assumptions that these proxy companies are pretty shady for the most part seem to be justified - they seem to be even more involved in scam, fraud, DDOS etc. than we discussed in the past.

The common saying "if you are not paying you are not the customer but the product" is not common enough yet. It is btw. also true for Cloudflare - if you are using their free tier you are the product and you pay with your data. Which may seem a fair deal for many but yet it is the same pattern: How many people have asked themselves with Cloudflare (or any other free product) why it is free and what may in fact be tied to it to pay w/o paying money?