Crazy amount of guests

[smallwheels]

couldn't we get the same list of residential proxies and block them?

Let's see: There are roughly probably at least 30 sellers of those proxies. Each typically claims to have several million IP addresses on offer. Those are dynamic addresses, so typically a physical proxy will change its IP every 24hours in most cases. So will have to sign up with all these sellers (which becomes expensive), find out all their proxy IPs (which you can't), repeat that every day and then feed them into your blocking list. More than a full time job I guess.

Some IP reputation services like Proxycheck.io do that btw.. That makes it possible for them to get insights about the proxiy-ips, the infrastructure of the proxy sellers and which proxy sellers share IP pools (which is very common). The outcome looks like i.e. that:



Yet they are only able to detect a fraction of a fraction of the existing residential proxies.

 

[smallwheels]

How can we consolidate the numbers reported by XF online users vs Google analytics real time users.
View attachment 347045View attachment 347046

You probably can't in a useful way. Most of the bots don't serve the Google Analytics script, so they don't show up in analytics.

 

[Alvin63]

Thanks - thought it sounded too simple. Definitely needs some legislation if this is all currently legal as residential proxies are being abused, and some people don't know they are one.

 

[Alvin63]

Had another idea. Out of the 1.37 million requests, 1.31 million of them are from desktop - and as they all hit Whats new then maybe I could just block anything using desktop and trying to access whats new, while whitelisting registered members and known bots. While this does run the risk of blocking a genuine guest from whats new, it's probably a low risk as most genuine users use mobiles.

So a block rule:

(cf.client.device_type eq "desktop" and http.request.uri.path contains "/whats-new/" and not http.cookie contains "xf_user" and not cf.client.bot)

Any thoughts? But then I suppose they'd just move to another path like posts or threads.

 

[Anthony Parsons]

Anthony,
I have watched this page with interest but overwhelmed with the amount of info. I have cloudflare account and have a few simple securities rules set up there.
What types of rules are you setting here?
Managed rules, custom rules, IP access rules?

Managed rules are cloudflare, primarily, upgraded accounts allows you to make changes and add CF managed rules to your site. Custom rules are what most people use when creating a rule for their website.

Thank you for the link about Turnstile - I don't fully understand the article - it sounds complicated.

Turnstile is more applicable to stopping human spam on your site. If you had a human spam problem, turnstile can solve that automagically for you by placing turnstile into every POST, for example. Instead of a page failing, XF reloads via ajax, on posting a cookie is presented to the browser. If its not a real person sitting there, fail. If it is, success. This stops spammers registering, posting a few times, getting past human moderation, then handing an account to software to go nuts and spam the site. Member, guest, turnstile just works behind the scenes.

Turnstile helps on things like registration, posts, searches, etc. It does not help on stopping bots scraping your site, that is a different problem.

I'm still a bit confused about the skip list. It includes Baiduspider - would I really want that? I've had it blocked in .htaccess for a long time. Also - I don't understand what "lower" in there means.

I posted earlier my actual list. https://xenforo.com/community/threads/crazy-amount-of-guests.233649/page-87#post-1796967
You questioned it here: https://xenforo.com/community/threads/crazy-amount-of-guests.233649/page-87#post-1797018
I responded with an extended list if you wanted global coverage... though yes, if you don't want China, Korea, Russia, etc, then go back to my original list of essentials.

A quirky thought, if these people building scrapers, can get a list of residential proxies, like on that video, couldn't we get the same list of residential proxies and block them?

No. Residential IP's change. I have already proven you can adequately block residential proxies by targeting header information. Saying that, if someone goes balls to the wall and completely spoofs a real browser, fingerprint, cookies, javascript, behaviour, combined with residential proxies, then its a sitewide managed challenge for every visitor pretty much. That would be the only way to stop it at that level. Right now... most aren't mimicking the full browser specifics needed, nor the behaviour. You can catch both easily and target it. If they fix that and stop being greedy, then it shifts to Very Difficult.

How can we consolidate the numbers reported by XF online users vs Google analytics real time users.
View attachment 347045View attachment 347046

Your realtime users, are probably more like 1-10 online at any given time.

Had another idea. Out of the 1.37 million requests, 1.31 million of them are from desktop - and as they all hit Whats new then maybe I could just block anything using desktop and trying to access whats new, while whitelisting registered members and known bots. While this does run the risk of blocking a genuine guest from whats new, it's probably a low risk as most genuine users use mobiles.

So a block rule:

(cf.client.device_type eq "desktop" and http.request.uri.path contains "/whats-new/" and not http.cookie contains "xf_user" and not cf.client.bot)

Any thoughts?

Yer, don't try and target requests. Desktop is not identification of a person or bot, xf_user does not authenticate a member either and you would block guest humans from whats new. Also, CF requests does not mean the request reached your server, it would record it from CF cache I believe. A request is not a user either, one page load can be 30 to 80 requests.

If you had 40 requests per page load, your requests, that is roughly 34k pages.

 

[Andy.N]

Is there any reliable prevention method against spike of bots that cause server load to go skyhigh?
My current server load is sitting at under 2 but from times to time, it will jump to 100 or more for 10-15 minutes and then go back to normal.
What are you guys using that reliably prevents this spike? Or this is the time we are living in and nothing we can do about the spike?

 

[Alvin63]

Is there any reliable prevention method against spike of bots that cause server load to go skyhigh?
My current server load is sitting at under 2 but from times to time, it will jump to 100 or more for 10-15 minutes and then go back to normal.
What are you guys using that reliably prevents this spike? Or this is the time we are living in and nothing we can do about the spike?

It's just a case of using the rules to reduce the numbers hitting the server and the site really.

I have three rules
1) Block rule - I have some countries blocked, some ASN's blocked and some bots blocked.

That did reduce the numbers a lot so the rest of the rules had less to deal with. But as they are residential proxies they just move worldwide to other countries. But blocking China, Malaysia and Myanmar were big ones for me and various ASN's in Singapore (don't block Singapore as a country as Google is there).

2 and 3 are Challenge rules - it depends where your bots are hitting. Mine are 99% hitting whats new. So I have a challenge rule for whats new, search, members, misc/cookies, threads, posts.

That stops most of them hitting those paths, except the sophisticated ones that can solve challenges.

To find out which paths your bots are hitting the most, you could try downloading a raw server log and uploading to Claude AI who will tell you which paths they are hitting the most, which countries are in there etc, and also what type of browser they are using.

I currently have 1.37 million requests, but only 900 guests on the site and server cpu only 2% because the rules are keeping most of them off the server. The bot swarm can fluctuate up and down.

Before the Cloudflare rules my server cpu was at 100% and they even took the site offline temporarily.

 

[Alvin63]

Is there any reliable prevention method against spike of bots that cause server load to go skyhigh?
My current server load is sitting at under 2 but from times to time, it will jump to 100 or more for 10-15 minutes and then go back to normal.
What are you guys using that reliably prevents this spike? Or this is the time we are living in and nothing we can do about the spike?

If you want to try just one custom rule to challenge the bots, you could try this if you want but if you read back a bit, Anthony has some good custom rules.

This one is fairly straightforward to achieve using the field boxes - but this is the expression. Note, my rule contains a "referer" - that means that genuine guests are not likely to get a Cloudflare challenge. However some bots can fake a referrer and get past it. But it still stops most of them for me. So you can just paste that into the expression box or set it up manually via the fields. If you want to just try one rule. This challenges the bots who try to hit/scrape whats-new, search, members, misc/cookies, posts, ams and threads. If you don't have ams it just won't do anything.

And set for Managed Challenge. If you have existing rules, they need to be in the right order. Block rules first, Challenge rules after that. Each path has
not http cookie xf_user - that means registered members don't get challenged, only bots.
not cf.client.bot - that means known bots like Google and Bing don't get challenged
http referer eq " " - that means the majority of genuine guests don't get challenged, just bots.

(starts_with(http.request.uri.path, "/whats-new/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/search/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/members/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/misc/cookies") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/posts/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/ams/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "") or (starts_with(http.request.uri.path, "/threads/") and not http.cookie contains "xf_user" and not cf.client.bot and http.referer eq "")

 

[Andy.N]

We are playing whack a mole for a seeable future it seems. Once we figure out something, bots will come up with new ways to attack.

 

[Alvin63]

We are playing whack a mole for a seeable future it seems. Once we figure out something, bots will come up with new ways to attack.

Blocking countries is whack a mole. Challenging forum pathways gets them wherever they are from. I still found it helped to block China and get the numbers down there to start with.

But it does seem to be the new norm. I had no Cloudflare rules at all until about 6 weeks ago and suspect I'll have to leave them there permanently now.

It's a battle between the bots and Cloudflare - hopefully Cloudflare will keep coming up with new ways to stop the bots if the bots get round some things.

 

[Anthony Parsons]

Is there any reliable prevention method against spike of bots that cause server load to go skyhigh?
My current server load is sitting at under 2 but from times to time, it will jump to 100 or more for 10-15 minutes and then go back to normal.
What are you guys using that reliably prevents this spike? Or this is the time we are living in and nothing we can do about the spike?

Yes, lock your server to cloudflare IP's, so nothing can touch your server IP directly, use authenticated origin pulls, then you can control what hits your server from CF. Without that, bots go round CF direct to your IP and do what they want. If on shared hosting: https://xenforo.com/community/threads/crazy-amount-of-guests.233649/page-62#post-1793025

Otherwise, Nuclear option by @BrettC is: Anubis.

 

[BrettC]

How can we consolidate the numbers reported by XF online users vs Google analytics real time users.
View attachment 347045View attachment 347046

Google Analytics is based on raw page access and invoking the GA script, similar to the forum active users list invoking any XF2 routes. However, GA will tend to intentionally undercount clients when the client intentionally ignores/filters/blocks the GA URL or has JS fully disabled.

Is there any reliable prevention method against spike of bots that cause server load to go skyhigh?
My current server load is sitting at under 2 but from times to time, it will jump to 100 or more for 10-15 minutes and then go back to normal.
What are you guys using that reliably prevents this spike? Or this is the time we are living in and nothing we can do about the spike?

If you must go the nuclear route, Anubis or some form of a WAF prior to the actual XenForo instance. Cloudflare will also fit the bill for this. @Anthony Parsons has posted some awesome CF rules that can be applied against your website.

Blocking countries is whack a mole. Challenging forum pathways gets them wherever they are from. I still found it helped to block China and get the numbers down there to start with.

But it does seem to be the new norm. I had no Cloudflare rules at all until about 6 weeks ago and suspect I'll have to leave them there permanently now.

It's a battle between the bots and Cloudflare - hopefully Cloudflare will keep coming up with new ways to stop the bots if the bots get round some things.

Blocking countries is an option if you need a knee-jerk hard-stop for temporary purposes. I wouldn't rely on country code level blocking in the slightest though - IPv4 and IPv6 ranges are all mishmashed and never 100% correct. For example, an Australian IP address and associated CIDR was redelegated to an ISP in the UK. You've now potentially blocked UK users when you meant to block AU users.

Otherwise, ASN blocks are excellent surgical methods to block toxic networks... but again, should not be relied upon - runs into the same issue as CC blocking. ASN's can always change CIDR's in use.

If at all possible, PoW challenges seem to be the best method to halting, or in the least, reducing, these ai-scraper swarms.

 

[Andy.N]

@Anthony Parsons
you need to put the wealth of knowledge in an article or guide resources for CF members. I have a paid account there and would like to utilize it more but hard to keep track of all the changing rules, etc.
May seem like a breeze to people who has been battling bots for the past few months but for the casual members who just got hit with this issue, it may be overwheelming.
Thanks for your service.

 

[Alvin63]

Well I got sick of the xxx ers hammering day and night the last few days - nearly all from USA. It wasn't particularly causing any issues on the server or in the guest list, but it was creeping up and up every day and it was bugging me that origin was now serving more than Cloudflare. So I decided to temporarily add this block rule. And see what happens. Just wanted to block them.

Requests went down from 1.4 million to 24k within minutes. Low risk of blocking a genuine visitor from Whats New (but not the rest of the site). So I'll see if the bots decide to move to a different path or give up and go away. They are virtually all using desktop and 99% hitting whats new.

I'll leave it temporarily anyway. Down from 1.4 million to this within 5 minutes



starts_with(http.request.uri.path, "/whats-new")

and (http.user_agent contains "Windows NT"

or http.user_agent contains "Macintosh"

or http.user_agent contains "X11")

and not http.user_agent contains "Android"

and not cf.client.bot

and not http.cookie contains "xf_user"

 

[Alvin63]

Claude just analysed the log - every single block was a bot - no real users. None had a referer. I guess I could maybe add a referer to reduce the low risk of a genuine guest being blocked when clicking on whats new.

 

[Andy.N]

Claude just analysed the log - every single block was a bot - no real users. None had a referer. I guess I could maybe add a referer to reduce the low risk of a genuine guest being blocked when clicking on whats new.

You should. What is the chance of real users being able to contact you via the contact form if they are being blocked entirely from your site.

 

[Alvin63]

From Claude server log analysis

	Before (13:06–15:29)	After (15:31–15:48)
Requests reaching your server	about 270 a minute	about 28 a minute
What’s New requests	about 46% of everything	1 request in 18 minutes
Service worker requests	about 17,500	almost none

That’s roughly a 90% fall in what your server has to handle, starting at 15:30. The service worker requests are the last step of the bots’ routine, so when they’re blocked at What’s New, that tail disappears too.

What’s left

The 497 requests since 15:31 look like normal background traffic. They come from Microsoft and Google addresses (probably Bing and Google fetching pages), a handful of crawlers, and ordinary UK visitors such as Virgin Media customers. I see no new path taking the swarm’s place yet. The single What’s New request that got through came from an IP that is probably a member, since it passed the cookie exemption.

Eighteen minutes is a short window. If they move, it may take hours or days to show, so I’d run the same check on tomorrow’s log.

The referer idea

You could add and not http.referer contains "thehamsterforum.com" to the rule, so that anyone clicking through from a page on your own site is let in. It would spare the genuine guests who click the What’s New link. I wouldn’t do it yet, for two reasons:

• Very few people benefit. Only about 3 of the 97 readers in the earlier log ever opened What’s New.
• It gives the swarm a one-line bypass. A referer is just a header, and 499 of the 500 events had none, so they’re not sending one today. If they ever copy yours, they’d walk past the rule.

If a member or guest tells you they were blocked, add it then. Until then, I’d leave it out.

 

[Alvin63]

You should. What is the chance of real users being able to contact you via the contact form if they are being blocked entirely from your site.

I checked the block page in an incognito browser and it does give a message saying this site is using security - contact the administrator etc. It only blocks whats new IF they are on a desktop. Not the rest of the site - they can still click on Home, Forum, Contact us etc.

But yes I think adding a referer is a good idea despite what Claude said.

 

[Alvin63]

Guest numbers down to 20 - just Robots and members. Even if I just do this temporarily it is giving a lot of satisfaction

 

[Alvin63]

The rule after this one is a challenge rule for search, members, misc/cookies, posts and threads, so if they switch to another path they'll just get challenged as before.