Crazy amount of guests

Rules haven't changed and they're in the right order - adding the fake browser rule just meant working out which type of challenge had priority but that's sorted now.

The levels now aren't affecting the server, I just want to keep the numbers down.

Rule 4 is the difficult one as the bots solve the challenge - so the rules above that help keep vast numbers out of rule 4.

Anyway some analysis tonight showed the majority of IP's in rule 4 are from US data centres. So just wondering if it's ok to block the ASN's of these data centres or if there could be any downsides. I guess it might block VPN users?

These were all hitting the site:
  • Leaseweb USA (ASN 393886, 395954, 394380, 27411, 7203)
  • HostPapa / ColoCrossing (ASN 36352)
  • FDCservers.net (ASN 30058)
  • EGIHosting (ASN 18779)
  • Colocation America Corporation (ASN 47007)
  • Web2Objects LLC (ASN 62874)
  • ISP4Life INC (ASN 394814)
  • Servers.com, Inc. (ASN 7979)
  • QuickPacket, LLC (ASN 46261)
  • WhiteLabelColo (ASN 394474)
  • Charles River Operation (ASN 64249)

Cloudflare numbers.webp
 
Humans don't have the same IP, and when they solve it once, they get x time without solving the next challenge based on your CF settings, default 30 minutes.
 
I just went to your site from AU and had to solve it, but I now have 30 minutes use before I have to solve it again. So every user is pretty much getting this with your site from all the challenges you have in place for x countries.
 
Based on your rules, you have 18% of humans getting caught in rule #4, whom are solving it and passing through. If you made that non-interactive, you would have the same result, except, people wouldn't have to click anything.
 
I had the logs analysed - bar a handful of humans they are all automated bots hitting whats new. And yes all countries except USA, Canada, Uk and most of Europe will now get an interactive challenge - except registered members. Which is ok. Australia has banned hamster keeping anyway so no members from there :-)
 
You had the logs analysed? Are you relying solely on AI for this? If you are, then you're in deep trouble. AI is a tool, it is not an authoritative source, it makes a lot of mistakes.

Go back to your AI analysis and ask it for the same information you have, except this time you want a 90% confidence score, and see how different the information it provides you is.

Note that is only asking for 90% confidence, not 100%. Ask that of the AI analysis and you will get a very limited result set.
 
Last edited:
Partially - I've also looked at the server logs and cloudflare logs myself. Yes they can make mistakes. But they are automated bots.
 
I appreciate that rule you gave me though, it is working very well and AI wouldn't have been able to do that, it can be good for analysing things though. :-)
 
I could potentially just lose rule 5 - it's only blocking 37,000 or so. But the rules work together. Country block and ASN blocks first - which are keeping a huge amount of numbers out. Followed by your rule which is getting all the ones with fake browsers. The rest in rule 4 are 98% automated and going direct to Whats-New/posts and scraping. The ones going to different forum paths bypass rule 4 and get challenged in rule 5. Registered members and guests in US, UK, Canada and Europe are whitelisted so don't get any challenges at all.

It just evolved under pressure over the last week and a half - I'm sure there are better ways.
 
A start is always a really good place... refine now.

Your first two rules can be done in one, is in country OR is in ASN. Rules work in order, so it hits rule one first, processes, is in country OR is in ASN, then moves to rule #2. Because the first two rules are both block, one rule can do the same thing.

The only rule I personally care about, is search protection, as that saves a server being taken out quickly.

Code: 
(starts_with(http.request.uri.path, "/whats-new") or starts_with(http.request.uri.path, "/search"))
and not cf.client.bot
and not http.cookie contains "xf_user="
 
Anthony Parsons said:
A start is always a really good place... refine now.

Your first two rules can be done in one, is in country OR is in ASN. Rules work in order, so it hits rule one first, processes, is in country OR is in ASN, then moves to rule #2. Because the first two rules are both block, one rule can do the same thing.

The only rule I personally care about, is search protection, as that saves a server today being taken out.

Code: 
(starts_with(http.request.uri.path, "/whats-new") or starts_with(http.request.uri.path, "/search"))
and not cf.client.bot
and not http.cookie contains "xf_user="
Click to expand...
That's what my rule 4 is. What's-new, search and members. It's whats-new that all the bots are going directly to and scraping.
 
The problem with all rules, is they come with unintended consequences. So whilst my browser impersonator rule will do its intended job, it will also have unintended consequences, like blocking Google Pagespeed, which is automation, but its an evaluation tool which has purpose. That is just one example, there will be more legitimate things it will block, not all automation is bad.
 
Welcome to the role of a decision maker, where every decision is a compromise between the different possible paths of consequences, and you cannot afford decision paralysis either.

Also, don't let "perfect" get in the way of "good".

Do the rules reduce abuse significantly, maybe at the cost of a few annoyed visitors? Good job, call it a win and move on with other things. Should you try to succeed at blocking 100,00% of the bots with absolutely zero impact on legitimate visitors? Not in our Universe, that's possible only in Utopia.
 
Anthony Parsons said:
This is why a VPS is really required, and it sucks IMO that is needed today to control this issue where hosts aren't doing the job for their own hardware.
Click to expand...

Managing a VPS in a responsible way is way over the top for many forum admins, judging from how many already struggle with the most basic things when admistrating XF configurations alone. Shared hosting on the other hand is typically chosen based on price by many if not most - so while I share your criticism in principle it is in the end a "you get what you pay for". Also, for a typical hoster it would be practically impossible to create a good level of protection for his customers as what they host vastly differs as does their audience. For a handful of $ per months it is not possible to create the individual protection that would be needed. A good managed server could be an alternative and clearly, current times would create a huge opportunity for a hosted XF offering or things like XF cloud - if they would provide a really good bot protection (and customers would be willing to pay for it).
 
Last edited:
I still prefer managed challenges. They are non-interactive for most human users, unless CF detects something questionable and only then do they present an interactive challenge for a human to solve. If anything, a non-interactive challenge lets through fewer genuine human visitors as it gives them no option to solve a challenge if their configuration won't automatically solve the challenge.

The way I've been reading things in this thread, managed challenges are assumed to be interactive, which isn't completely true.

developers.cloudflare.com

Interstitial Challenge Pages

Full-page challenge screens that verify visitors before they reach your website.
developers.cloudflare.com developers.cloudflare.com

Actions
The following challenge types are the available actions when you create a WAF rule for a Challenge Page.


Non-Interactive Challenges
With a Non-Interactive Challenge, Cloudflare makes the determination on whether or not the visitor is automated based on the limited information attained from their browser signals via an injected JavaScript. Then, it presents a Challenge Page that requires no interaction from a visitor except the JavaScript processed by their browser.

The visitor must wait until their browser finishes processing the JavaScript, which typically takes less than five seconds.

If the visitor passes the challenge, the original request continues to the destination URL. If the challenge fails or cannot be completed, the visitor is presented with another interstitial Challenge Page.


Managed Challenges
Managed Challenges are where Cloudflare dynamically chooses the appropriate type of challenge served to the visitor based on the characteristics of a request from the signals indicated by their browser. This helps avoid CAPTCHAs ↗, which also reduces the lifetimes of human time spent solving CAPTCHAs across the Internet.

Most human visitors are automatically verified and the Challenge Page will display Successful. However, if Cloudflare detects non-human attributes from the visitor's browser, they may be required to interact with the challenge to solve it.

Cloudflare recommends Managed Challenges for most WAF rules. Unless there are specific compatibility issues, do not use other challenge types.


Interactive Challenges
Interactive Challenge Pages require a visitor to interact with the challenge to pass.

Cloudflare always recommends using a Managed Challenge. For more information, refer to the Cloudflare blog post ↗.
Click to expand...

Also: https://blog.cloudflare.com/end-cloudflare-captcha/#many-solutions-not-one

Essential quote:

After our non-interactive challenges have been run, we evaluate the gathered signals. If by the combination of those signals we are confident that the visitor is likely human, no further action is taken, and the visitor is redirected to the destined page without any interaction required. However, in some cases, if the signal is weak, we present a visual puzzle to the visitor to prove their humanness. In the context of Managed Challenge, we’re also experimenting with other privacy-preserving means of attesting humanness, to continue reducing the portion of time that Managed Challenge uses a visual puzzle step.
Click to expand...

I figure, do whatever you prefer. I'm sticking to managed.
 
Wildcat Media said:
I still prefer managed challenges. They are non-interactive for most human users, unless CF detects something questionable and only then do they present an interactive challenge for a human to solve. If anything, a non-interactive challenge lets through fewer genuine human visitors as it gives them no option to solve a challenge if their configuration won't automatically solve the challenge.
Click to expand...
I too think that a non-interactive challenge yields false negatives. And not only do visitors not have to checkmark anything, they can't. So genuine visitors can't prove they are in fact human. I use a managed challenge for paths that members, known bots and genuine visitors need to have access to. Works well. I had a swarm come in last night around 10pm and they stayed the entire night: 271.83k requests. An awful lot hitting /cookies. But look how nicely my managed challenge is dealing with those: 202.92k requests and a CSR of 0.03%. And visitors on my forum are normal. I'm happy.

Scherm­afbeelding 2026-09-16 om 18.03.53.webp
 
Levina said:
I too think that a non-interactive challenge yields false negatives. And not only do visitors not have to checkmark anything, they can't.
Click to expand...
I had a couple of members overnight say that they were blocked, and I'm not sure what they mean by that, as the only thing I had blocked were certain CF verified bots and known bot user agents. Anything else in my rules (country, ASNs, etc.) are challenged, not blocked. So that led me to believe that they were presented a challenge the browser couldn't solved, and were denied access. I've since changed those back to a managed challenge.

Frustrating because one of them was a staff member, and it looks as though he was trying to access via a phone. Had he been on the computer, I allow the xf_user cookie in my whitelist and he should have had no issues. The frustrating part is he uses Comcast, and they use IPv6 addresses which change every time, so I can't even trace back into the logs to see what happened.

I did change my robots.txt with the recent updates but those don't affect human users.
 
You must log in or register to reply here.

Similar threads

O
  • Question Question
XF 2.2 Huge amount of guests
Replies
6
Views
189
infinitysnapz
infinitysnapz
Wildcat Media
Unusually high number of guests (4x to 5x normal)
Replies
8
Views
222
Tumpe
Tumpe
Tyson
Possible to limit amount of threads a guest can view?
Replies
4
Views
1K
Junior
J
Back
Top Bottom