Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
You could throw a pretty decent party with that amount of guests.But only 600 guests on my forum - which is good
What amount of guest used to be normal on your forum before the bot waves started, so probably 1,5 - 2 years ago?I'm still wondering about these numbers. This sounds even worse than the worst times here on the XF forums back in spring this year. Way worse. Are these bots or are these rather requests? But even then: Your forum is decent, but in comparison to others on here not that huge:
How old is it?
Im comparison: Mine is four years old and has roughly half as many messages, double as many threads and four times the number of users. The largest number of guests on the forum I ever saw was 1800 in spring last year during a bot wave from Hetzner. I have started to block bad requests already two years ago and never stopped since. The highest number of IPs that tried to reach my forum within a 24h period was something like 12.000 and clearly, more than 10k has been an exception that I can count on one hand. Normal over the last months has been 1.200 - 2.500, atm bots seem to get nasty again, so numbers between 3.000 and 4.000 are kind of normal atm with about 7.500 being the peak over the last weeks. Most of them do never reach the forum anyway and are blocked upfront completely.
So either bots like hamsters way, way more than my forum topic or there must be another difference that makes my forum less attractive for bots than others. I mean: I clearly won't complain about that
- just wondering what the reason may be for this immense difference.About 180 on average.You could throw a pretty decent party with that amount of guests.
Maybe they haven't caught up with you yet! I think they are just targeting Xenforo forums maybe. Mine is linked all over the internet. Members are only since 17 months ago when I started from scratch again after closing it down in March 2025 due to the OSA. But then reopened again in May 2025. Prior to closing down it had 850 members but started again with about 40. So it's gone from 40 to about 1100 in the last 17 months. But all the posts were still there as I had left the site in tact when it was taken offline.
So Forum is 4 years old but membership only 17 months old. It's been building up nicely. But yes indeed what is the interest in my little forum? I don't think the bots are particular - it's just whatever they find. Although I did wonder at first if it was because I had affiliate links in home page articles.
Anthony, can I ask what's the first rule you have there?
I block cf.client.bots, because if you look at the approved bot list for cloudflare, it does not mean the bots are good, it just means the bots belong to, basically, a business who has registered their bot with cloudflare versus a person running their own bot to do anything they want. This rule allows only the absolute essential bots onto my site, with skip, the second rule outright blocks cf.client.bots, so anything not in the first rule skipping further rules, is caught. There is over 700 CF approved bots. 90% of them are rubbish to most websites.
Code:
cf.client.bot and (
lower(http.user_agent) contains "googlebot"
or lower(http.user_agent) contains "bingbot"
or lower(http.user_agent) contains "applebot"
or lower(http.user_agent) contains "duckduckbot"
or lower(http.user_agent) contains "bravebot"
or lower(http.user_agent) contains "yandexbot"
or lower(http.user_agent) contains "baiduspider"
or lower(http.user_agent) contains "yeti"
or lower(http.user_agent) contains "petalbot"
or lower(http.user_agent) contains "oai-searchbot"
or lower(http.user_agent) contains "claude-searchbot"
or lower(http.user_agent) contains "perplexitybot"
or lower(http.user_agent) contains "chatgpt-user"
or lower(http.user_agent) contains "claude-user"
or lower(http.user_agent) contains "perplexity-user"
or lower(http.user_agent) contains "facebookexternalhit"
or lower(http.user_agent) contains "linkedinbot"
or lower(http.user_agent) contains "twitterbot"
or lower(http.user_agent) contains "telegrambot"
or lower(http.user_agent) contains "discordbot"
or lower(http.user_agent) contains "slackbot"
or lower(http.user_agent) contains "bingpreview"
or lower(http.user_agent) contains "microsoftpreview"
or lower(http.user_agent) contains "googleimageproxy"
)Edit - yes it's request numbersI'm still wondering about these numbers. This sounds even worse than the worst times here on the XF forums back in spring this year. Way worse. Are these bots or are these rather requests? But even then: Your forum is decent, but in comparison to others on here not that huge:
View attachment 346901
How old is it?
Im comparison: Mine is four years old and has roughly half as many messages, double as many threads and four times the number of users. The largest number of guests on the forum I ever saw was 1800 in spring last year during a bot wave from Hetzner. I have started to block bad requests already two years ago and never stopped since. The highest number of IPs that tried to reach my forum within a 24h period was something like 12.000 and clearly, more than 10k has been an exception that I can count on one hand. Normal over the last months has been 1.200 - 2.500, atm bots seem to get nasty again, so numbers between 3.000 and 4.000 are kind of normal atm with about 7.500 being the peak over the last weeks. Most of them do never reach the forum anyway and are blocked upfront completely.
So either bots like hamsters way, way more than my forum topic or there must be another difference that makes my forum less attractive for bots than others. I mean: I clearly won't complain about that
Today it's up to 1.32 million! While numbers on the site are up a bit it's not drastic - about 900 ish guests fluctuating.
Thanks!
Thanks to the recent updates to IP Threat Manager I've now a much clearer picture. I.e. the last 24hs look like that:
Quite some activity with bots coming in waves.
67% of requesting IPs have been blocked, most based on ASN. At other times, more are blocked based on country, that changes dynamically.
As usual, most baddies come from the US:
To be honest: I am honestly thinking about blocking the US as a country as it is the major source of bots an is not in my core target group anyway.
Either way, only very little is left fort the Proxycheck.io API:
Those additional few found by proxycheck were all resident proxies from either UK, US, Canada, Spain, Portugal or Germany. Distribution was as follows:
This means, subtracting members and allowed bots from the 1.425 IPs that Threat Manager has led pass we are talking about ~500 guests, an unknown amount of which are resident proxies.
After IP Thread Manager traffic bumps into Bot Guard and there the 24h window looks like that:
Which means 212 of those 500 guests were stopped by bot guard's captcha. Makes ~290 guests left. Additionally, some may be stuck in the browser verification and not reach the captcha. Some of leftover guests may be genuine (so happy to serve them) while an unknown amount targeted a non existing URL or something protected by login - those who get a 404, 403 or similar are ignored by Bot Guard anyway. From experience this is quite a lot of the bots, i.e. when they try to hit a picture directly or a member profile (both quite common). What goes through only sees the first post of a given thread, so cannot cause not much damage. If I should guess I'd say probably around 50-100 guests at best will able to see the (for them limited) forum content and almost all of them are probably genuine.
Overall, I am quite happy with this setup. There are some things that can be improved further (which mainly needs more data drilling to find out how exactly), but overall it works pretty decently as protection layer.
Quite some activity with bots coming in waves.
67% of requesting IPs have been blocked, most based on ASN. At other times, more are blocked based on country, that changes dynamically.
As usual, most baddies come from the US:
To be honest: I am honestly thinking about blocking the US as a country as it is the major source of bots an is not in my core target group anyway.
Either way, only very little is left fort the Proxycheck.io API:
Those additional few found by proxycheck were all resident proxies from either UK, US, Canada, Spain, Portugal or Germany. Distribution was as follows:
This means, subtracting members and allowed bots from the 1.425 IPs that Threat Manager has led pass we are talking about ~500 guests, an unknown amount of which are resident proxies.
After IP Thread Manager traffic bumps into Bot Guard and there the 24h window looks like that:
Which means 212 of those 500 guests were stopped by bot guard's captcha. Makes ~290 guests left. Additionally, some may be stuck in the browser verification and not reach the captcha. Some of leftover guests may be genuine (so happy to serve them) while an unknown amount targeted a non existing URL or something protected by login - those who get a 404, 403 or similar are ignored by Bot Guard anyway. From experience this is quite a lot of the bots, i.e. when they try to hit a picture directly or a member profile (both quite common). What goes through only sees the first post of a given thread, so cannot cause not much damage. If I should guess I'd say probably around 50-100 guests at best will able to see the (for them limited) forum content and almost all of them are probably genuine.
Overall, I am quite happy with this setup. There are some things that can be improved further (which mainly needs more data drilling to find out how exactly), but overall it works pretty decently as protection layer.
Last edited:
So basically, are those the ones you allow via a skip rule? And then no not.cfclientbot in any further rules? Also, with such a long list of 700 or so, how to know which might actually have be helpful if blocked. Eg are all SEO bots unhelpful? Are all Data collection bots unhelpful? (This seems to include Pinterest). Are all Monitoring and Operations bots unhelpful? This seems to include Google API, Shopify and Github.
It looks a good idea, but I think I'll pass on it, because it wouldn't really make a difference to my residential proxy bots, which are the vast majority of bots hitting my site.
I had Claude check my server logs and only 19 of Cloudflare's 700 verified bots visited my site, in a 24 hour period, so maybe it isn't an issue.
Last edited:
Is there a way of being able to use Turnstile in custom rules, instead of Managed, or Interactive challenges? This article shows how these bots are solving challenges (grr) and actually teaches people how to bypass Cloudflare. But it does say they can't solve Turnstile.
There are a number of articles like this on the internet - to help bots bypass Cloudflare challenges and fingerprinting. This one shows it's not just puppeteer but addons to it.
www.browserless.io
Here's another one
humanbrowser.cloud
There are a number of articles like this on the internet - to help bots bypass Cloudflare challenges and fingerprinting. This one shows it's not just puppeteer but addons to it.
Bypass Cloudflare with Puppeteer (2026 Guide) – Scrape Protected Sites
Struggling to scrape sites protected by Cloudflare? This 2026 Puppeteer guide walks you through stealth setup, proxy rotation, CAPTCHA detection, and how to fal
www.browserless.io
Here's another one
Puppeteer with Residential Proxy in 2026: Complete Setup Guide
Set up a residential proxy in Puppeteer in 2026. Working code examples, country selection, fingerprint matching. Stop getting blocked by Cloudflare and DataDome.
I still use my own Block Specific Bots rule but added Anthony's skip rule (now the first rule). It's really nice and I am very happy with it.
No amount of human written text is enough to train AI on; if your website contains a decent amount of it, you are a target. The more, the larger the target.
I looked into my high amount of guests and found they were hammering /misc/style-variation
Added the following to CloudFlare and 6k blocked within minutes and the unusual guest traffic is gone. No issues with normal users changing style variations.
Added the following to CloudFlare and 6k blocked within minutes and the unusual guest traffic is gone. No issues with normal users changing style variations.
Code:
starts_with(http.request.uri.path, "/misc/style-variation") and not any(http.request.headers["x-requested-with"][*] eq "XMLHttpRequest")See: https://xenforo.com/community/threads/crazy-amount-of-guests.233649/page-83#post-1795269
GPT categorisation of CF allowed bots.
Integrating Turnstile with the Cloudflare WAF to challenge fetch requests
By editing or creating a new Turnstile widget with “Pre-Clearance” enabled, Cloudflare customers can now use Turnstile to issue a challenge when a page’s HTML loads, and enforce that all valid responses have a valid Turnstile token
From quickly skimming it, it uses pre-clearance mode.
If you wanted to get a little broader, fairly safely, with my rule, you could expand to:
Code:
cf.client.bot and (
lower(http.user_agent) contains "googlebot"
or lower(http.user_agent) contains "googleother-image"
or lower(http.user_agent) contains "bingbot"
or lower(http.user_agent) contains "applebot"
or lower(http.user_agent) contains "duckduckbot"
or lower(http.user_agent) contains "bravebot"
or lower(http.user_agent) contains "yandexbot"
or lower(http.user_agent) contains "baiduspider"
or lower(http.user_agent) contains "yeti"
or lower(http.user_agent) contains "petalbot"
or lower(http.user_agent) contains "slurp"
or lower(http.user_agent) contains "mojeekbot"
or lower(http.user_agent) contains "kagibot"
or lower(http.user_agent) contains "oai-searchbot"
or lower(http.user_agent) contains "claude-searchbot"
or lower(http.user_agent) contains "perplexitybot"
or lower(http.user_agent) contains "duckassistbot"
or lower(http.user_agent) contains "mistralai-index"
or lower(http.user_agent) contains "chatgpt-user"
or lower(http.user_agent) contains "claude-user"
or lower(http.user_agent) contains "perplexity-user"
or lower(http.user_agent) contains "mistralai-user"
or lower(http.user_agent) contains "meta-externalfetcher"
or lower(http.user_agent) contains "facebookexternalhit"
or lower(http.user_agent) contains "linkedinbot"
or lower(http.user_agent) contains "twitterbot"
or lower(http.user_agent) contains "telegrambot"
or lower(http.user_agent) contains "discordbot"
or lower(http.user_agent) contains "slackbot"
or lower(http.user_agent) contains "bingpreview"
or lower(http.user_agent) contains "microsoftpreview"
or lower(http.user_agent) contains "googleimageproxy"
or lower(http.user_agent) contains "whatsapp"
or lower(http.user_agent) contains "pinterestbot"
)Anthony,
I have watched this page with interest but overwhelmed with the amount of info. I have cloudflare account and have a few simple securities rules set up there.
What types of rules are you setting here?
Managed rules, custom rules, IP access rules?
I have watched this page with interest but overwhelmed with the amount of info. I have cloudflare account and have a few simple securities rules set up there.
What types of rules are you setting here?
Managed rules, custom rules, IP access rules?
Thank you for the link about Turnstile - I don't fully understand the article - it sounds complicated.See: https://xenforo.com/community/threads/crazy-amount-of-guests.233649/page-83#post-1795269
GPT categorisation of CF allowed bots.
View attachment 347029
Integrating Turnstile with the Cloudflare WAF to challenge fetch requests
By editing or creating a new Turnstile widget with “Pre-Clearance” enabled, Cloudflare customers can now use Turnstile to issue a challenge when a page’s HTML loads, and enforce that all valid responses have a valid Turnstile tokenblog.cloudflare.com
From quickly skimming it, it uses pre-clearance mode.
If you wanted to get a little broader, fairly safely, with my rule, you could expand to:
Code:cf.client.bot and ( lower(http.user_agent) contains "googlebot" or lower(http.user_agent) contains "googleother-image" or lower(http.user_agent) contains "bingbot" or lower(http.user_agent) contains "applebot" or lower(http.user_agent) contains "duckduckbot" or lower(http.user_agent) contains "bravebot" or lower(http.user_agent) contains "yandexbot" or lower(http.user_agent) contains "baiduspider" or lower(http.user_agent) contains "yeti" or lower(http.user_agent) contains "petalbot" or lower(http.user_agent) contains "slurp" or lower(http.user_agent) contains "mojeekbot" or lower(http.user_agent) contains "kagibot" or lower(http.user_agent) contains "oai-searchbot" or lower(http.user_agent) contains "claude-searchbot" or lower(http.user_agent) contains "perplexitybot" or lower(http.user_agent) contains "duckassistbot" or lower(http.user_agent) contains "mistralai-index" or lower(http.user_agent) contains "chatgpt-user" or lower(http.user_agent) contains "claude-user" or lower(http.user_agent) contains "perplexity-user" or lower(http.user_agent) contains "mistralai-user" or lower(http.user_agent) contains "meta-externalfetcher" or lower(http.user_agent) contains "facebookexternalhit" or lower(http.user_agent) contains "linkedinbot" or lower(http.user_agent) contains "twitterbot" or lower(http.user_agent) contains "telegrambot" or lower(http.user_agent) contains "discordbot" or lower(http.user_agent) contains "slackbot" or lower(http.user_agent) contains "bingpreview" or lower(http.user_agent) contains "microsoftpreview" or lower(http.user_agent) contains "googleimageproxy" or lower(http.user_agent) contains "whatsapp" or lower(http.user_agent) contains "pinterestbot" )
I'm still a bit confused about the skip list. It includes Baiduspider - would I really want that? I've had it blocked in .htaccess for a long time. Also - I don't understand what "lower" in there means.
A quirky thought, if these people building scrapers, can get a list of residential proxies, like on that video, couldn't we get the same list of residential proxies and block them? I mean I know that would block real users but a block page could have a message saying contact us if you're not a bot or something. And to be honest if real users are being used as residential proxies, maybe they deserve to be blocked! I know some of them don't realise they are but ....... But I guess it wouldn't be a full worldwide list.
