Crazy amount of guests

probably a more sophisticated behavioral tracking could be of interest to better identify bad actors. Bot Guard has done a lot of positive in regards of identifying baddies based on their properties, but a behavioural risk analysis, fully integrated into XF, learning from the patterns of the individual forum and it's visitors and members would be another level (but be vastly complicated and possibly only feasible by XF itself).

Just on this point - Bot Guard has started now down this path. In one of the more recent release, Chronology and sampled registered member behaviour begins to serve as a signal for bot behaviour. The first entrant into the space was link chronology and sampled dwell time. So the navigation must be realistic given the page the bot was on, and, the amount of time it spent on the page must be realistic given the real human behaviour on the forum.
 
Another thing that was mentioned earlier was the draft web bot auth spec. Bot Guard will introduce a beta support implementation for the draft spec (as it seems Google is trailing it, likewise Cloudflare, so I might as well throw in support). I like the idea as an additional way to verify an acceptable bot. It also provides a way for much smaller / bespoke bots who don't own IP ranges and whatnot, to be able to cryptographically sign and verify the bot origin. Sounds good to me.
 
Chronology and sampled registered member behaviour begins to serve as a signal for bot behaviour.

Regarding challenges: Today I saw this:
Bildschirmfoto 2026-06-24 um 09.20.54.webp
The thread in question is a couple of years old, relatively short, has no newer posts and is typically visited occasionally, but very rarely. 271 hits on a single day to this thread is clearly a pattern that stands out. It is not always as visible as here but strangely enough bots often do visit the same odd pages in parallel in numbers. This should be a signal that could be spotted and used easily - the only challenge is to avoid false positives when for some reason real traffic in numbers hits a buried thread.

Of the other routes challenged I would for most of them had guessed "no normal traffic" as well for most of the visits. Turned out to be true.

The result: For said thread not a single visitor passed the captcha. :p

Bildschirmfoto 2026-06-24 um 09.22.55.webp
 
Regarding challenges: Today I saw this:
View attachment 340458
The thread in question is a couple of years old, relatively short, has no newer posts and is typically visited occasionally, but very rarely. 271 hits on a single day to this thread is clearly a pattern that stands out. It is not always as visible as here but strangely enough bots often do visit the same odd pages in parallel in numbers. This should be a signal that could be spotted and used easily - the only challenge is to avoid false positives when for some reason real traffic in numbers hits a buried thread.

Of the other routes challenged I would for most of them had guessed "no normal traffic" as well for most of the visits. Turned out to be true.

The result: For said thread not a single visitor passed the captcha. :p

View attachment 340459

Yep... "pressure aware routes" is a feature coming soon... :)
 
In a related piece of news, looks like Cloudflare also had it with bots and the continuous chase to outsmart captcha-bypassing tools, and will try a new angle by involving Mozilla Firefox, Google Chrome and Microsoft Edge in a partnership towards identifying human traffic. The press release is from two days ago, and was echoed in many channels - here's one:


The details are not yet clear, but apparently the browsers will be able to differentiate people vs bots and then present some sort of "token" to other sites that confirm the visitor is not a bot. We'll see how it goes; it will take some time until it gets implemented though.
 
The details are not yet clear, but apparently the browsers will be able to differentiate people vs bots and then present some sort of "token" to other sites that confirm the visitor is not a bot. We'll see how it goes; it will take some time until it gets implemented though.

It will be interesting to see when this comes out. I can hardly imagine it would work. Many scrapers actually use genuine browsers. I know mine did.
 
I also have my doubts this scheme will bear fruit.

@smallwheels, i hear you about the two layers needing to be integrated.
Thanks for describing your setup and it's effectiveness.
My system is now starting development.

Today I had some spam bots get through.
And I smiled.
What a classic problem from the past to deal with quickly instead of my site being rinsed by crawlers.

I've been having that too. Got all the stock options turned on, but it's not enough.
We're going to use cleantalk because it's been bad. If that doesn't work, we are building AI spam filtering!
 
I believe XF Bot guard uses that internally.
There's also a library called thumbmark, but i don't know how good it is.

Yes, the open source version which is available here: https://github.com/fingerprintjs/fingerprintjs

Important to note for those unaware; in of itself, it is not a bot detection. It is just a ... "somewhat" reliable signal of "we may have seen this browser before" - which Bot Guard uses to glue together sessions, even though the IP may have changed, cookies wiped, etc.
 
In a related piece of news, looks like Cloudflare also had it with bots and the continuous chase to outsmart captcha-bypassing tools, and will try a new angle by involving Mozilla Firefox, Google Chrome and Microsoft Edge in a partnership towards identifying human traffic. The press release is from two days ago, and was echoed in many channels - here's one:


The details are not yet clear, but apparently the browsers will be able to differentiate people vs bots and then present some sort of "token" to other sites that confirm the visitor is not a bot. We'll see how it goes; it will take some time until it gets implemented though.
The problem that I feel with this is that Cloudflare is attempting to do what Google did with the mass promotion of Chrome over Firefox and IE from way back in the day: market capture. In this instance, CF's implementation for user v. bot is likely going to only work between browsers that support it and cloudflares' network. If it's made fully open source, so that developers can integrate it on their own self-hosted hardware/networks, it could likely catch on and become a hit. However, i'm left with more doubts on the idea.

https://github.com/antifraudcg/proposals/issues/22 - I'm iffy on the whole idea.
 
The problem that I feel with this is that Cloudflare is attempting to do what Google did with the mass promotion of Chrome over Firefox and IE from way back in the day: market capture. In this instance, CF's implementation for user v. bot is likely going to only work between browsers that support it and cloudflares' network. If it's made fully open source, so that developers can integrate it on their own self-hosted hardware/networks, it could likely catch on and become a hit. However, i'm left with more doubts on the idea.

https://github.com/antifraudcg/proposals/issues/22 - I'm iffy on the whole idea.

The downside of this proposal, IMO, is that it shifts the burden onto the user to already have some kind of accepted “trust relationship” with the web.

Not necessarily money, although paid subscriptions are one obvious example, but some scarce signal: accounts in good standing, phone numbers, email addresses, subscriptions, long-lived relationships with services, etc. In other words, your ability to move around the web with less friction starts to depend on whether you can present credentials from issuers that other sites are willing to trust.

That feels future-blind to me, because AI agents and scrapers can also operate through real browsers, real accounts, real sessions, and real paid infrastructure. So we are not really proving “human”. We are proving “this traffic is backed by a scarce credential and is currently under a rate limit.”

That may help with volumetric abuse, sure. But it also creates a class problem. A user with paid services, old accounts, phone-verified identities, and mainstream browsers will naturally look more trustworthy than a user who avoids those things. Privacy-focused users, users with fresh accounts, people using niche browsers, VPNs, Tor, self-hosted tools, or independent agents may end up looking suspicious by default.

The proposal says it wants open participation, and that any organisation could theoretically provide credentials. But in practice, sites and Moderators will choose which Anchors they trust. That is where the gatekeeping risk lives. Big platforms, browser vendors, telcos, payment providers, Cloudflare-style anti-abuse networks, and major account providers become credible trust sources. A startup, a small forum, or a random XenForo community probably does not.

So the web slowly becomes: “bring a credential from an issuer we like.”

That is the part I dislike. It feels like lazy solution design: make the victim/user jump through legitimacy hoops, rather than forcing the bot/abuser to prove itself through behaviour, intent, cost, or consequence at the point of abuse.

This is basically the opposite direction to how I think bot defence should work. I would rather assess behaviour and risk at the edge of the actual request than build a web-wide trust economy where the already-established players get to define who starts from a position of trust.

The crypto may be clever. The incentives are the problem.
 
Interesting. Seems the open source is not as reliable as the commercial... offcourse, money talks, BS walks, and all. Unbelievable these companies.

It's hard to say for sure, but, I do think it is likely a lot of the 'reliability' is tooling built on top of the open source version of it, rather than instead of or branched from the open source version. Similar in a way that I have built tooling on top of it in XF Bot Guard.

However, they may very well also be holding out the most delicious features from the open source product. Can't discount that potential.
 
the web slowly becomes: “bring a credential from an issuer we like.”
This is exactly where we're going.

Today, even CF has difficulties to identify humans and bots based just on the traffic passing through. Bots are learning fast to simulate all parameters that will soon make them impossible to distinguish from real people. User agent, source IP, cookies, referral, speed of clicking through links, the use of an actual hidden browser window to have exactly the right browser capabilities footprint - one by one these are becoming no longer reliable individually or even put together.

It has to be taken further upstream onto the devices themselves, and not only look at how the browser (bot or human controlled) is interacting with ONE domain, but if that browser is used like a human would use it across multiple sites. This is sensitive and you can't just start checking for cookies from other domains; you need the browsers to cooperate and hint that yes, the patterns of web access through the browser are consistent with a human doing their daily stuff, or no, this browser is going randomly to one URL and another, as if it was instructed by a bot.

And honestly this is the smallest step foward that still attempts to safeguard privacy and anonimity online. Because if the people running the bots keep escalating this fight for taking it all, the defense mechanisms will also keep going up. I fear that one of the realistic scenarios for the future is the one where you have to insert your issued ID in a slot in the computer and authenticate biometrically with your digital certificate before you can do anything online. That's the end of the anonymous internet as we know it, and the battle will then be put in the camp of stealing or faking IDs. And I'm not sure if it would be better if that ID was government-issued or private company issued; companies can capture and sell your metadata for services accessed, government can turn into a police state... 1984 here we come.
 
That's the end of the anonymous internet as we know it

Well, you basically pinned it there. That's why I don't like it. I am not some privacy geek, but, I don't think we should deny humans any privacy just because of "bots".

Not only is that lazy and distasteful design IMO - but it is easily "won" by AI using browsers like a human ... which is where scraping is going already. Heck, I own aiwebscraper.com and had built a fully AI-capable-driven scraper, before deciding this was not the good fight.

I feel like this approach being discussed is akin to taking a shot to prove you are human. Not to bring vaccines into this, but to prove a point: it is the wrong end of the stick IMO.
 
Back
Top Bottom