That is a fair point, and as I mentioned earlier, it was a suggestion that may or may not be helpful. It's an option. Your summary of options was also helpful. However, it can be overwhelming for people not knowing where to start. It was for me at first. So just sharing what helped (and much of mine was achieved by help from others on here).
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
I still don't get where these high numbers come from. Your forum is about the same age as mine, I have more four times of your members and double the threads, you have double the posts of my forum. Overall relatively comparable.
I don't count requests but IPs that try to connect which is obviously a way lower number and therefor looks way less impressive.
On an average, calm day we are talking about 1.600 - 2.500 IPs within 24h and ~50% of them are deflected.As I had exactly the nice number of 2000 this morning I did some screenshots for a breakdown. Here's what it looks like:
Absolutely no load on the system, a simple rule set, barely any bot or scraper get's trough. My genuine traffic comes to 99% from the DACH area, hence these countries are strong and relevant. With a different audience it would look differently. Yet still it is easy to identify bad ASNs and countries worth blocking and no issue to do so.
Proxyckeck barely finds anything harmful that would be not already blocked, for the most part residential proxies here and there that then gain a temporary block automatically.
Life is good.
Last edited:
I don't think it's the size of the site that matters - the bots just hit any site - they seem to have a penchant for Xenforo forums. I did wonder at first if it was because my site has affiliate links as I got hit around the time I posted an article with affiliate links in on the home page. But that may have been a coincidence, because I'd had affiliate links for a while.
There are different swarms and botmasters out there I guess and there is one really massive swarm and 30% of that one can solve Cloudflare challenges.
Those numbers on my site now are really low compared to the 1.4 million requests a day until recently. Once I closed all avenues they seemed to give up mostly. Not completely. Plus I relaxed things a bit in some areas to minimise friction for real guests.
Maybe the number of requests is more than the number of IP's but I suspect it's not much different, because the bots go through 3 IP's in about 2 seconds. They switch IP's fast to avoid rate limiting presumably. The same request can have 3 or 4 IP's.
There are different swarms and botmasters out there I guess and there is one really massive swarm and 30% of that one can solve Cloudflare challenges.
Those numbers on my site now are really low compared to the 1.4 million requests a day until recently. Once I closed all avenues they seemed to give up mostly. Not completely. Plus I relaxed things a bit in some areas to minimise friction for real guests.
Maybe the number of requests is more than the number of IP's but I suspect it's not much different, because the bots go through 3 IP's in about 2 seconds. They switch IP's fast to avoid rate limiting presumably. The same request can have 3 or 4 IP's.
Do you have any indictaor for that? Sounds like confirmation bias to me. They seem to scrape whatever they can, all kinds of forums, website, blogs - simply everything that does not lock them out, regardless of the content.
Surely there are different players in the game but I think it are not too many overall, less than three digits that use residential proxies massively, as a silly wild guess, based on nothing but gut feeling.
Maybe that's the secret ingredience. I started blocking radically more or less two years ago, way earlier than most that now suffer from immense swarms. Maybe early enough that a bunch of players have already given up on my forum early.
One single pageload can easily consist of 40-80 requests. The scrapers often do not load the full page (which is one way to identify them), so they often will have way less than that and as many rotate to a different IP/proxy after one try the average number of requests per IP for scrapers via proxies will surely not be in the hundreds. But there is no formula to extrapolate from one into the other.
That's technically impossible.
Today's hits have been coming from Cox Communications (ASN 22773) using the latest version of Chrome (157) on Linux, and mostly accessing random images in /data and /styles. This one has been annoying since Cox is used by a ton of legitimate people and I don't want to inconvenience our Linux users by challenging their latest Chrome version. 
I challenge all Linux users, mainly because the forums I manage are far from tech-oriented and we have so few visits from logged-in members using any type of Linux that it's a non-issue. (Most members are understanding about why we need to filter things, and are accepting of any challenges they might face before they log in. Not all forums have considerate members like this.) With the xf_user and xf_session cookie detection in the filter, once they log in they won't be challenged any further. If I had a more tech-oriented forum, I certainly wouldn't challenge any Linux users.
I wonder if they rotate through IPs. In other words, use an IP once or a few times, rotate through dozens? hundreds? of others, then start back at the beginning with the same list. If they are targeting one ISP to use as their proxy, there have to be limited IP addresses available that have the backdoors available to them.
The question I'm wondering about is...how do all of these Cox customers have back doors to the bots? I realize a lot (or most?) of it is through installing apps with hidden terms and conditions (including apps that use a particular SDK that has the back doors built into it--I'd have to find the article again), but are there that many Cox customers who have done this? And why Cox? With all the major ISPs we have here, is there something about Cox that is easier for bots to access than, say, AT&T Fiber, Spectrum, Xfinity, etc.? Or, are these going to be next?
Also, on which devices? We can't know for sure, but we do know there has been hidden activity on things like TVs and IoT devices in the home, as well as computers, tablets, phones, etc. that may also have questionable software installed.
I don't need answers to any of these--it's just thinking out loud. But it's just so danged frustrating for all of us that have to deal with the swarms.
I'm actually considering doing an advisory thread on the forums I manage that gets into some of these issues, and to not trust things like janky VPN providers that supply the backdoor to the bots through the proxies. Yet...which VPN providers can even be trusted?
Whatever is happening lately, they're outpacing Cloudflare and everyone else in finding ways to get through to sites.
Yeah it would be far less of a problem for me if the excluding the xf_user cookie from challenges didn't leave things like avatars and other images broken once the user has logged in. They don't get challenged which is correct, but the images remain busted. I've made many attempts to troubleshoot that, but never got anywhere with it, so I have to consider that this will still impact logged in users in some way. It's frustrating.
I wouldn't mind answers... its a mystery. I know there are millions of resi proxies available, the sites outline it as their selling point to use their service. That is fairly large scale infection, let alone all those who accept terms or such for their ip to be used.
That is the exact same problem I've been having on one of my sites, and the only difference is that the site with the broken avatars, etc. uses Cloudflare R2 buckets. I've asked in other threads and nobody has ever replied about having the same issue, or having any idea what is causing it. The public R2 bucket uses a subdomain of the forum (I use forumdata.xxxxxxx.com). The only thing I can think of is perhaps whitelisting images, but that leaves them open for abuse unless I put those exceptions as part of the whitelist rule that looks for either or both cookies.
If I find anything out, I'll have to post here and tag you. I'm bookmarking this post so I can come back to it.
This also makes me wonder if I have ever inadvertently downloaded anything suspicious. But, I really don't use anything out of the ordinary these days, and most software I use is either purchased or subscribed to, with the rest being long-available programs like PuTTY, HeidiSQL, etc. I try to stick to well-known apps on my other devices, but who's to say that something doesn't have a back door in it. I'm not so sure about my Ubuntu desktop and Ubuntu media server in the house, but I've done my best to lock down my home network as best I can.
And I agree...it's all a mystery, and "they" are continually one step ahead of us. Frustrating. Think of all the hours we all have wasted. Sure I'd like some answers, but nobody here among us "good guys" can answer them.
My hosting provider (MDDHosting - my review) made some bot protection that integrates with Turnstile.
Works wonderfully for both Wordpress and Xenforo. I have been using Cloudflare proxy, but this has been a lot more efficient (works along with Cloudflare now). No complaints from human visitors, and a very low number of bots and bandwidth usage due to scraping (it was getting pretty high before I implemented the hosting provider's extra protection).
Not sure how exactly it was built, hope they'll make a public blog post or something explaining it in detail, and I suppose most folks could build something similar for their own servers.
Works wonderfully for both Wordpress and Xenforo. I have been using Cloudflare proxy, but this has been a lot more efficient (works along with Cloudflare now). No complaints from human visitors, and a very low number of bots and bandwidth usage due to scraping (it was getting pretty high before I implemented the hosting provider's extra protection).
Not sure how exactly it was built, hope they'll make a public blog post or something explaining it in detail, and I suppose most folks could build something similar for their own servers.
I just did some more research and actually feel like it may be related to the fact that my subdomain where XF images are hosted (data.website.com) do not typically have a cookie applied to them. I've checked my Network tab in the browser's Inspect section, and do not see any cookie showing when inspecting the broken images, which may mean that the skip rule does not take into account any images on this subdomain and thus visitors still need to pass a CF challenge for them to not be broken. According to the Cookie docs section of XF's guide:
Config.php options | XenForo Documentation
All of the information XenForo needs to connect to your database server is located in the src/config.php file.
docs.xenforo.com
I may try setting this. It will cause everyone to need to log in again, but perhaps the cookie will apply then and things will be fine. Maybe you can also give this a shot?
My best guess would be a standard router with a compromise that has not been patched. Either because they are old routers that Cox has not replaced or it's just not a published vulnerability. Or maybe the ISP subscription comes with some other IoT device that has some hole in it or something dubious installed out of the box - maybe some kind of set-top-box type thing, etc. Given the numbers we see from Cox I think it must be something a bit more universal than the individual customers have all installed the same "whatever" on a device at home.
We have bots hitting our image URLs directly as well, so excluding those paths from challenge rules is not something I can do sadly!
If those images are cached at the edge, it doesn't matter how they hit the path, they get the cached version, not your origin pull and load. That is factoring CF and your server locked to its IP's, no way to use your IP/image-path
Not right, and you're not dumb... we're all just trying to work this crazy sh*t out is all.
I've been testing CF's cloudflared tunnel, and whilst it all sounds good, when you throw 5000 rps at it, cloudflared consumes near 1.5 cpu cores just to exist. That is insane for the minor benefits it provides vs locking to CF IP's and using origin pulls.
