Crazy amount of guests

But basically the list of verified bots in the updated addon I use for registration request, might not be the same as Cloudflare's verified bots anyway?
It's likely not to be. An addon and Cloudflare don't use the same list of bots. Cloudflare's list includes bots approved and listed by them, within categories so we can block bot categories if we want to.

The way Anthony's filter works is similar to mine. In a nutshell:

Select the good bots you want to allow on your site, and check them using cf.client.bot to make sure that, for example, they are not a bot masquerading as Google. So it's an AND rule, like: your list of bot referrers) AND (cf.client.bot) in other words. (Or vice versa.) Those get the SKIP rule to whitelist them.

Then in a later rule, all the rest of those bots (several hundred) are blocked.

The (lower(http_user_agent) ... sets all the incoming referrer strings to lowercase for comparison purposes. (I use it so I can "wildcard" things like "google" in the user agent string and not have to worry about capitalization.)
 
It's likely not to be. An addon and Cloudflare don't use the same list of bots. Cloudflare's list includes bots approved and listed by them, within categories so we can block bot categories if we want to.

The way Anthony's filter works is similar to mine. In a nutshell:

Select the good bots you want to allow on your site, and check them using cf.client.bot to make sure that, for example, they are not a bot masquerading as Google. So it's an AND rule, like: your list of bot referrers) AND (cf.client.bot) in other words. (Or vice versa.) Those get the SKIP rule to whitelist them.

Then in a later rule, all the rest of those bots (several hundred) are blocked.

The (lower(http_user_agent) ... sets all the incoming referrer strings to lowercase for comparison purposes. (I use it so I can "wildcard" things like "google" in the user agent string and not have to worry about capitalization.)
Thanks. I'm still not sure how the block fake bots rule fits in with it - or does it become part of it? I have that separately at the moment.
 
While not related to the AI Scraper hellscape... has anyone noticed in recent months that bad actors are using Google Cloud's infrastructure, and has been on an absolute rampage performing fairly notable DNS probing/rainbow-attacks? That is then coupled with persistent, continuous, and distributed attacks on a web server to said DNS queries that have an answer for. For whatever it is worth, I am talking about random IP addresses from all of their CIDR's on Google Cloud being involved with this episode.

I've added a new action within Anubis and have reconfigured Anubis to reject (444) clients, not a deny (403) or allow (200) to these ranges. I probably need to issue a pull request with this modification onto the main Anubis branch, as it seems to be working without issue. :-)
 
I'm still not sure how the block fake bots rule fits in with it
What fake bots? Either a bot is a bot, or it's not. (say that ten times fast)

If you're talking about my browser impersonator rule, that is not catching fake bots, it's catching bots. It is catching unregistered bots that private individuals have created, and are using to create havoc upon the Internet.

It is catching bots trying to fake being human, pretending to be a human visitor to your site from a web browser, BUT, they aren't a real web browser at all, they're a bot with fake browser header credentials. No real browser is missing real browser specifics.
 
Thanks. I'm still not sure how the block fake bots rule fits in with it - or does it become part of it? I have that separately at the moment.
As @Anthony Parsons stated above: It's either a bot, or it's not.

When we refer to bots, we're effectively depicting the entire ecosystem of bots. That would include both good and bad bots. That includes, but isn't limited to entities such as indexer bots, AI Scrapers, probing bots/drones, automated impersonation bots, etc. That's just the tip of the iceberg when you look at it from a holistic view point though. It's a whole genre of 'things'.

The other side is just... the human. Human do smart thing, human do big dumb thing. Humans are unpredictable with whatever thing that they choose to do. Whereas, bots, bots tend to have patterns/similar or repetitive actions. ;)
 
has anyone noticed in recent months that bad actors are using Google Cloud's infrastructure
I experienced issue with bots hosted by Google Cloud earlier this year. But I got smae issue also with AWS, Oracle, and Microsoft clouds.
 
What fake bots? Either a bot is a bot, or it's not. (say that ten times fast)

If you're talking about my browser impersonator rule, that is not catching fake bots, it's catching bots. It is catching unregistered bots that private individuals have created, and are using to create havoc upon the Internet.

It is catching bots trying to fake being human, pretending to be a human visitor to your site from a web browser, BUT, they aren't a real web browser at all, they're a bot with fake browser header credentials. No real browser is missing real browser specifics.
Hi yes I meant impersonator - just different language sorry :-). I've been calling it the fake bots rule to myself - they fake being google etc when they're not. I think that regardless of how many bots there are, to me it's not just about stopping them it's about knowing the robots who declare themselves on site are who they say they are. Hope that makes sense.

I was meaning in terms of rule wording - I'll try and read back and find it - I've been using your rule for the impersonators for a while. And was thinking of using the skip rule idea and blocking cf.client bot, but wondering how that affected the rule for the impersonator bots, in terms of syntax etc :-)

It's currently at the end of my first block rule - so this is the end part of the rule (the rest is blocking ASN's and a few countries). As I had added some things I wasn't even sure if I'd got that right.

I actually have meta just blocked regardless. The rest of them are just blocked if they're not cf.clientbot as is the intention I think.

So what I was asking was

a) Have I got the syntax right in this rule (brackets etc) now I've altered it (it maybe needs tidying up as well) and
b) How would this rule work If I then blocked cf.client bot in the same rule, with a skip rule before it? Would it stay the same or would it need to be altered?

Eg 1) Skip rule for chosen verified bots 2) Block all cf client bots (to stop the rest as you have done) 3) Impersonator bots rule - and presumably that would only now include the verified bots from the skip rule?

or (lower(http.user_agent) contains "meta-externalagent") or (lower(http.user_agent) contains "meta-webindexer") or (lower(http.user_agent) contains "meta-externalads") or (lower(http.user_agent) contains "deepseekbot") or (http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot" or http.user_agent contains "Google-InspectionTool" or http.user_agent contains "AdsBot-Google" or http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot" or http.user_agent contains "PerplexityBot" or http.user_agent contains "CCBot" or http.user_agent contains "facebookexternalhit" or http.user_agent contains "Twitterbot" or http.user_agent contains "DeepSeekBot") and (not cf.client.bot)
 
Pop it into AI and it will tell you if the expression is correct or not, and be specific with the AI what CF plan you're on, as what you can do in an expression changes per plan.

You can do a lot of things with a CF expression, but you should go read the CF docs on how expressions work, ie. AND will be processed before OR.

Don't try and complicate an existing rule, because an existing rule may be simpler written when you add something to it, changing everything you had to a different logic.

Code:
(
  (
    http.user_agent contains "GPTBot"
    or http.user_agent contains "ClaudeBot"
    or http.user_agent contains "PerplexityBot"
    or http.user_agent contains "Google-InspectionTool"
    or http.user_agent contains "AdsBot-Google"
    or http.user_agent contains "CCBot"
    or http.user_agent contains "facebookexternalhit"
    or http.user_agent contains "Twitterbot"
    or http.user_agent contains "DeepSeekBot"
  )
  and not cf.client.bot
)
or
(
  cf.client.bot
  and (
    http.user_agent contains "GPTBot"
    or http.user_agent contains "ClaudeBot"
    or http.user_agent contains "CCBot"
    or http.user_agent contains "DeepSeekBot"
  )
)

If BLOCK, then the top half blocks pretenders and the bottom half would block legitimate. Change it to SKIP and they do the same thing for SKIP action, one does one thing, the other half does the opposite.

My Point: Adding things that do the opposite, in the same rule, can get complicated.

Keep It Simple Stupid is my principle I like to live by, especially with CF rules. Less is often more.

If you use AI to help you, ask it to make the rule do what you want, in the shortest, simplest way, possible, using what your plan has available to it.

It's like the list of 700+ bots on cf.client.bot, if you only want to use 50, then its easier to allow 50 and block the rest, not block 650+ and allow the rest.
 
I experienced issue with bots hosted by Google Cloud earlier this year. But I got smae issue also with AWS, Oracle, and Microsoft clouds.
Nowadays it is very easy to automate the spin-off of a new virtual server on a cloud provider, deploy whatever bad intended tools on it, and forge that server until something catches it and turns it off. Rinse and repeat.

I was looking last week through logs randomly and found thats some bots, identifying (as user agent and source IP) from several of the big AI services, were attempting repeatedly to exploit possibly misconfigured websites and make it display environment variables or default files with secrets for cloud services or API calls, which most likely is a very successful strategy for the plethora of idiotic vibe coders out there. Is this related to what they are currently describing in media as "rogue agents" which, oopsies, "uncontrollably" hacked foreign government services again? Back in my days it didn't matter that they were autonomous agents or people - the company's leadership would have been held accountable for the hack, just like they would pursue hackers which broke into their private systems. Nowadays we're not far from being ordered to apologize to (random AI tool) for hurting its feelings by calling it a bad actor. Ffs...
 
I experienced issue with bots hosted by Google Cloud earlier this year. But I got smae issue also with AWS, Oracle, and Microsoft clouds.
That is not new but the question of @BrettC was specifically regarding DNS attacks:
bad actors are using Google Cloud's infrastructure, and has been on an absolute rampage performing fairly notable DNS probing/rainbow-attacks?
 
That is not new but the question of @BrettC was specifically regarding DNS attacks:
Indeed. These sort of attacks indeed have been around since the internet began, but the rapidity and distributedness(sp?) of these style of attacks is growing concerning. I normally see it as a "spray n' pray" or a blind method of attack - not this persistent and repeated level of attack.

The majority of these attacks begin at the DNS probe, and some attacks hit so fast that named/Bind's RRL is slow to react and kick in. Most of my domains do not have DNS wildcarding active or configured - which limits the scope of these attacks in the first place. However, wildcarding is used on a couple older domains that I have, and they're set to effectively honeypot-and-log bad actors with an actual IPv4 / IPv6 answer to the "non-existent" subdomain names. Which is leading me to see some of the more focused / targeted attacks on HTTP/HTTPS. These clients will not only attack the answered-to fake-subdomain name but also the non-subdomain domain.

The majority of these attack vectors begin at the DNS mass probing 'state', then the same IP range on Google Cloud slams my servers with large to massive amounts of random GETs and POSTs of varying attack vectors in an incredibly short amount of time. While Anubis, nginx, and other items are auto-filtering out most of this noise (and annoyances), the rate of attacks performed has increased a significant amount in recent months. I am legitimately unsure if this is some sort of newer-generation of AI-based mass probe attacks or some skiddy spinning up random GCloud virtual/spot servers across the GCloud infrastructure using some sort of script or C2.

I sure don't have this level of issue with AWS, among other cloud providers. Sure, they originate from other providers as well, but not to this extent that I am seeing with Google Cloud services.

The bots behavior is very similar to how the AI-scraper bots function: block/rate-limit the attackers IP, CIDR, or even ASN, and it swaps to another IP address quickly to continue probing/attacking. Which makes me ponder the idea that it's some sort of AI-based mass probe attacks over a general script-kiddy.
 
Pop it into AI and it will tell you if the expression is correct or not, and be specific with the AI what CF plan you're on, as what you can do in an expression changes per plan.

You can do a lot of things with a CF expression, but you should go read the CF docs on how expressions work, ie. AND will be processed before OR.

Don't try and complicate an existing rule, because an existing rule may be simpler written when you add something to it, changing everything you had to a different logic.

Code:
(
  (
    http.user_agent contains "GPTBot"
    or http.user_agent contains "ClaudeBot"
    or http.user_agent contains "PerplexityBot"
    or http.user_agent contains "Google-InspectionTool"
    or http.user_agent contains "AdsBot-Google"
    or http.user_agent contains "CCBot"
    or http.user_agent contains "facebookexternalhit"
    or http.user_agent contains "Twitterbot"
    or http.user_agent contains "DeepSeekBot"
  )
  and not cf.client.bot
)
or
(
  cf.client.bot
  and (
    http.user_agent contains "GPTBot"
    or http.user_agent contains "ClaudeBot"
    or http.user_agent contains "CCBot"
    or http.user_agent contains "DeepSeekBot"
  )
)

If BLOCK, then the top half blocks pretenders and the bottom half would block legitimate. Change it to SKIP and they do the same thing for SKIP action, one does one thing, the other half does the opposite.

My Point: Adding things that do the opposite, in the same rule, can get complicated.

Keep It Simple Stupid is my principle I like to live by, especially with CF rules. Less is often more.

If you use AI to help you, ask it to make the rule do what you want, in the shortest, simplest way, possible, using what your plan has available to it.

It's like the list of 700+ bots on cf.client.bot, if you only want to use 50, then its easier to allow 50 and block the rest, not block 650+ and allow the rest.
Thank you very much. Much appreciated. I don't think I'd 100% trust AI with code or rule expressions though. It's ok to analyse a log but it can get things wrong sometimes. And you'd get two AI's saying different things.

So an experienced human still knows more IMO :-)

Yes I had/have a lot of different things combined in one block rule due to only having 5 rules on the free plan. I finally decided to upgrade to the pro plan though, as I think I'm going to need these rules permanently, and I wanted a user-friendly html page instead of the block page on one rule - just in case. So one more forum cost - although I had just reduced some other costs this year so it's back to where it was.
 
However I did use a "fingerprinting" rule Claude did so I hope that's correct. It's a challenge rule for some Uk hosting sites ASN's and a smaller group of rp's that go to paths other than what's new and has particular browser patterns. About half are stopped by the challenge - the other half solve it - but it's quite a small group so not many getting through at all. Does it look about right? There's a chance a member could get challenged but real users aren't challenged anywhere else on the site (unless they're from certain countries I wouldn't normally have members from). But yes I'll checkout some Cloudflare documents when I get chance.

((ip.src.asnum in {212238 212286 30058 204139 210906 204646 203346 203020 9009 3257}) or (http.user_agent contains "X11; Linux" and http.user_agent contains "Chrome/154" and not http.user_agent contains "CrOS") or (http.user_agent contains "Macintosh" and http.user_agent contains "Chrome/142")) and not cf.client.bot and not http.cookie contains "xf_user"
 
I was looking last week through logs randomly and found thats some bots, identifying (as user agent and source IP) from several of the big AI services, were attempting repeatedly to exploit possibly misconfigured websites and make it display environment variables or default files with secrets for cloud services or API calls, which most likely is a very successful strategy for the plethora of idiotic vibe coders out there.

I've seen these kinds of attacks twice recently in logs and these bots are both numerous and seem to be pretty good at doing the evil.
In neither cases did they get close to landing an attack because:
  • i patch on the same day i hear of a security patch
  • i add other layers of protection on against classes of exploits, IE the kinds of things OWASP would recommend.
  • i also have fail2ban monitoring for excessive 403/404, password cracking attempts, and other probing.

Before and after AI, a lot of sysadmins don't bother being proactive like this because whoever hired him wants to spend $0 on security.
They need something to happen to get their boss/client to care and spend money.

What's even more bizarre is that some of my clients that have been hacked before i came on have seen the cost of it, yet still won't pay $100 to prevent a future $5,000 problem.

My systems are secure only because i figured out how to make it inexpensive to do proactive things and also usually successfully arm wrestle a client into paying for the work.

Thank you very much. Much appreciated. I don't think I'd 100% trust AI with code or rule expressions though. It's ok to analyse a log but it can get things wrong sometimes. And you'd get two AI's saying different things.

So an experienced human still knows more IMO :-)

I've been programming since the 90's.
I get better results writing regular expressions with AI than doing them by hand because it's a small part of what i do and never manages to get memorized.
It's faster for me to have the clanker write the regex to my specifications and thoroughly test it.
AI bots change their attack patterns faster than i can write security mechanisms by hand so it's essential that the process is speedy when a site is under duress.

I also have AI write me all kinds of specialized log analysis tools which allow me to narrow down what's going on very rapidly.
I just have to think out the design & test it - it gets me a result easily 10x faster.

I'm confident in using AI for these things because i have enough experience to know when it's gaslighting me or doing something stupid.
You can make up for that lack of experience with a lot more testing in this area.
 
Our ratio of members to visitors is heading upwards, and I noticed this interesting analytic chart on Cloudflare:

1792423249098.webp

Notice that at a point at about 1:00 PM on October 16 where managed challenges are bypassed, and how much of a leap it took. No rules were changed in the past week. Our stats for our origin access haven't changed, so it's not affecting the server much that I can see. A slight rise in the origin, but the amount served by Cloudflare did jump at the same time.

1792425088910.webp



Unrelated question....if unwanted bots "solve" a challenge, has anyone noticed if they are they solving interactive or non-interactive challenges the most, or is this something that we can't see in our Cloudflare analytics?
 
Unrelated question....if unwanted bots "solve" a challenge, has anyone noticed if they are they solving interactive or non-interactive challenges the most, or is this something that we can't see in our Cloudflare analytics?

I found that Interactive Challenge is marginally more effective than Managed Challenge but doesn't make a huge difference. If they can solve the challenge.
 
It's been a week since I set up the desktop/whats-new block and all still working really well so I've decided to keep it. It's blocked nearly all the vast numbers of residential proxies - ie blocked tens of thousands of requests a day. The only rp's left are a small handful that go to different pathways and most of those can't solve challenges - so the whats-new ones are the ones using real browsers that can solve challenges,and none of them carry a referer. Other rules challenge some data centres and most of those can't solve challenges either. So I've consistently only had real members, real guests and robots on my site for a week, with the occasional 2 or 3 bot IP's appearing briefly.

As mentioned before - this rule is probably site dependent - size of forum, what devices most people use etc. My site is quite small and virtually all members and guests seem to be using mobile, not computer - whereas 99.9% of the bots are using desktop. However there are some members who use computer or tablet so it has to allow for those. Despite the site being small - it still had vast numbers of bots hitting it.

Although the block rule exempts registered members with xf_user and has a referer for real guests - as pointed out earlier (thank you), there is a (low risk for me) low risk that a member might have whats-new bookmarked and be logged out and they'd get a block (just for that page, not the entire site).

So to mitigate that I have this html page now instead of the block page. I admit it would be annoying for a member or guest to be redirected in this way, but I think the chances of anyone hitting this page are extremely low. I've had logs checked daily for that rule and no real guests or members have been blocked on the whats-new page.

Adding the html page doesn't seem to have changed the bot behaviour. They are all just blocked and it's stayed that way.

This replaces the standard block page - but the rule is still a block for what's new if accessed directly using mac, windows or linux (ie via a bookmark or an automated bot going direct). There are no links to click - or the bots might click them.

Whats-new block custom page.webp

I still have a few countries blocked that I could have potential guests or new members from (Thailand, Phillipines, Mexico and Brazil) but can't unblock those as they are some of the worst for rp's. So I've done another "friendly" html page for those blocked countries (which might give a hint to genuine people to use a vpn if they want to access the site).

Country block html custom page.webp

I got claude to do the html pages, which needed to be less than 2kb.
 
Last edited:
Screenshot 2026-10-19 at 11.16.52 PM.webp
Massive uptick in US-based spam traffic in the last few days, particularly from ASN 401560. They've really been hammering away. Geez.
 
View attachment 347747
Massive uptick in US-based spam traffic in the last few days, particularly from ASN 401560. They've really been hammering away. Geez.
One Cable Network LLC - it's been high in residential proxy traffic along with 398781 Occulus. One Cable bots on my site were using real up to date chrome browsers.

If using cloudflare you could put 401560 in a challenge rule on its own and see what the solve rate is - about 30% of my rp's could solve challenges. But it did stop 70% of them.
 
Last edited:
Back
Top Bottom