Crazy amount of guests

[chillibear]

modern bots can slip past CDN-level checks. That’s exactly why the application-layer approach outlined by ES Dev Team is becoming increasingly important.

Totally agree. Were it not for the quantity and speed of the AI bot "browsing" and the subsequent load it'd not get flagged for the most part. Whilst annoying I tend to only cull the visits that cause heavy load at present whilst we (like ES Dev Team) ponder updating some of our code (also using Clickhouse FWIW) to be a little more intelligent. It does seem at least for now they are still going for direct requests so an analysis of traffic for a given "AI scraper" doesn't look like a normal user at least. However given the photos you see online of automated mobile phone farms and of course conventional headless browsers I'd not be surprised if we saw more stuff that is indistinguishable from normal traffic (except maybe in the speed). However I suppose making those extra requests for JS/Images/CSS and so forth must add up on the scraper's side so maybe they'll stick with what they have. It's actually been quite quiet on the scraping front for us at least the last week or two, just the normal better behaved bots.

I'd be quite interesting if XF internally had more sense of "usage checking" to identify real visitors from bots and so forth. I did start writing some statistical analysis code (outside of the XF codebase) for users at the start of the year to idly see if we might use it to highlight suspicious accounts, but alas "real work" got in the way and I've not gotten back to it yet.

 

[smallwheels]

I'd not be surprised if we saw more stuff that is indistinguishable from normal traffic (except maybe in the speed).

Depending from the audience of your forum/website geographics may also be able to be an indicator. If i.e. I get a sudden rush of visits from countries that I normally get barely visits from this may indicate a bot wave. Even more if they come from the same IP-Range/ASN. Also I see the URLs that the are visiting. If very old threads are visited all of a sudden from a bunch of guests at once (all visiting the same threads) and these come from unusual locations I can be pretty sure these are bots. There are a lot of behavioral patterns in bot traffic that aggregated can be used to identify them. Unfortunately many of them are specific to the website/forum, so for the most part nothing cloudflare could do automatically.

It's actually been quite quiet on the scraping front for us at least the last week or two

same here. A bunch of residential proxies in various countries at smaller scale, but nothing massive that I would have recognized since weeks.

 

[Tamer]

We’re seeing the same trend. For now the volume and request speed still make the scraping traffic identifiable, but that won’t last. That’s why behaviour-based filtering at the application layer is becoming essential beyond what a CDN can handle. @chillibear