Yeah I’ve had Occulus on my challenge list for a while, but it’s only recently that One Cable has started being so aggressive. I had them on block but just moved over to challenge and will see how it goes
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Actually maybe it would be better blocked - I just checked and I have it blocked - my logs for it (yours might be different) showed 98% requests were from Linux Chrome (full real browser), mainly going to whats-new and they carried a referer for the page they were requesting so they bypassed various rules due to the referer (that might not matter for your rules, I had a referer in my whats-new rule). And I think some were solving challenges as well. From what I've read it's unlikely to have real users but some people using VPN's via it could get blocked. I think my rationalising at the time was to not worry about the odd real user using a vpn. But risks vary with different sites maybe.
Last edited:
No. In this thread there are excellent recepies how to block those guests.
It won't.
If y'all haven't seen it yet, check out this thread by @LogicWhisper :
This is a very interesting but yet a bit alarming development. Barring the potential legalities and privacy issues from what they have going on over there... AI scrapers/bots/clients piggy backing off of legitimate users via a local browser is downright concerning. It is a vector that had been living rent free in the back of my head for a while now.
While I am definitely not a fan of the AI scraping turned mass mirroring of content, that sort of tool has been around for decades, such as HTTrack.
So now we have...
1. AI Scrapers from random sources -> easily observable and blocked/filtered correctly with a basic WAF configuration.
2. General Index & AI Training scrapers -> generally easy enough to identify who these are by ASN and UAs.
3. Legit client-side piggy-backing AI collection entites -> ???
I'm not sure how to deal with item #3 correctly. As that's now dealing with legitimate clients on a website, using a browser that is 'AI-Enabled' - for lack of better words. One could rate-limit a client thread-view rates, but that can quickly get into masses of false positives, which would require some serious log monitoring to finding a sweet spot. That in its own regard is extremely time consuming.
This is a very interesting but yet a bit alarming development. Barring the potential legalities and privacy issues from what they have going on over there... AI scrapers/bots/clients piggy backing off of legitimate users via a local browser is downright concerning. It is a vector that had been living rent free in the back of my head for a while now.
While I am definitely not a fan of the AI scraping turned mass mirroring of content, that sort of tool has been around for decades, such as HTTrack.
So now we have...
1. AI Scrapers from random sources -> easily observable and blocked/filtered correctly with a basic WAF configuration.
2. General Index & AI Training scrapers -> generally easy enough to identify who these are by ASN and UAs.
3. Legit client-side piggy-backing AI collection entites -> ???
I'm not sure how to deal with item #3 correctly. As that's now dealing with legitimate clients on a website, using a browser that is 'AI-Enabled' - for lack of better words. One could rate-limit a client thread-view rates, but that can quickly get into masses of false positives, which would require some serious log monitoring to finding a sweet spot. That in its own regard is extremely time consuming.
It is an interesting problem and one worth monitoring and discussing - however: I think it is way outside of the scope of this thread. The thread has already become almost unreadable and impossible to follow or draw useful information from it due to the enourmous amount of noise in the last weeks. It has become more of a personal diary full of very shallow short-lived things rather than the very stratetic and insightful debate it used to be earlier. It won't get better if we load even more topics into this thread that have nothing to to with the topic "enourmous amount of guests". I think we would be better served with a dedicated threat for that - that already exists and that you have linked to.
Agreed. It's likely time for a collaborative page of resources, tips, and tools on how to 'deal with and mitigate' AI Scrapers/botnets. This thread has been very noisy since most of us have found methods that can adequately manage abusive scraper bots, that being on-forum addons, same-server external-to-XF WAFs (Anubis/PoW), and external-server external-to-XF (Cloudflare).
Detailing those all into an organized flat-page wiki (heck, host the source on github, export it and format automatically to a site) would be the utmost ideal at this point. We've all not seen too much variance at this point in time.
I know this is just an example. Whilst this may work on some basic stuff, or create a basic replica of public content, it can't do as it claims: It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.
It can't walk server side folders and databases.
I don't think this poses any different of a challenge if you're blocking at DNS level. Site level, probably. Both if they've got fully fledged headers, JS, etc, and can impersonate an actual browser. The HTTrack one can't from a quick look.
I would think Anubis would also deal with this without interaction?
Anubis would (well, it should anyways) see it as a new session, as it's lacking a cookie with a validation string, thus throwing it a new challenge issuance. The backend / Anubis-side would/should also see a mismatch.
Haven't people already figured out ways around Anubis?
That's what I hoped - they would just go away - but they seem to be hitting all forums and the numbers can start to increase.
As it's quite a long thread, the general mitigations people have been using are cloudflare rules or, if no cloudflare, there are other addons or tools that others know more about than me.
Do you mean you're using Xenforo Cloud? If so you can still use the Cloudflare free plan. A few cloudflare rules would at least start to get the numbers down. The bots are probably residential proxies so come from all over the world so it's not always as simple as just blocking a country. It can also mean blocking some ASN numbers (eg from Singapore data centres).
But a start is setting up a rule to challenge various paths on the forum like whats new and search, as bots (in my case at least) seem to go direct to paths like that.
Yes. i sometimes see periods of bots breaking through the protection, but never overwhelming the system anymore.
The combination of fail2ban and anubis is quite good and we are typically meeting or exceeding the protection level provided by cloudflare without having to use a wall of cloudflare rules.
Another observation. While looking at one particular group of IP addresses accessing us via HTTP/1.0 (very few in number), they were hitting us in bursts of six or seven accesses per second, and each one was three different URLs. Two of the URLs were the home pages of the site (www.xyz.com and forums.xxx.com) while the third was for a different thread each time.
I don't know if any of the server-side solutions like Anubis or any of the XF add-ons would rate limit accesses (or at least flag them as suspicious), but it's pattern I see with others--the hits can be for a few different pages in the forum (threads, home page, etc.) all during a very short period of time, like a matter of seconds.
Cloudflare does have very basic rate limiting (on the free account level) but it won't catch this kind of logic, such as, accessing different resources on a site within a very short burst of time.
I think my rules may have caught these specific IPs with a managed challenge, but it's another pattern to watch for with other bots. No human can visit three different page views on a forum within one second's time.
I don't know if any of the server-side solutions like Anubis or any of the XF add-ons would rate limit accesses (or at least flag them as suspicious), but it's pattern I see with others--the hits can be for a few different pages in the forum (threads, home page, etc.) all during a very short period of time, like a matter of seconds.
Cloudflare does have very basic rate limiting (on the free account level) but it won't catch this kind of logic, such as, accessing different resources on a site within a very short burst of time.
I think my rules may have caught these specific IPs with a managed challenge, but it's another pattern to watch for with other bots. No human can visit three different page views on a forum within one second's time.
It's what the residential proxies do to avoid rate limiting IMO. They circumvent anything that could stop them in that way - except a cloudflare challenge. For the ones that solve a challenge, it's a problem - depending on how high the percentage is of the challenge-solvers. The challenge solvers aren't stopped by browser header rules either as they use full Chrome browsers and look like real people.
I liked one of the comments in the video - "wouldn't this (the t-pot software I assume) be good to find all the hacked devices of unknowing people and tell them". But I guess the software only shows IP's and locations, not names and addresses!
Even so there should be some kind of public information push to tell people with smart devices etc, how to reset them to remove any rp agreement (or whatever is installed in them). Is that actually possible? Re-setting to remove installed software?
It's still very weird to think of someone's smart washing machine or fridge being part of a botnet and attacking Xenforo forums!
Last edited:
As an update - if anyone is interested or it's helpful at all (feel free to ignore @smallwheels): So, the residential proxies have started evading my block rule - predictable I suppose, but as it had worked for 12 whole days I had begun to hope it was a solution. They suddenly appeared again yesterday - instead of whats-new they are now going to /goto/ and /threads/ direct. The same swarm of rp's - none carry a referer, all using up to date real browsers and 30% solve rate again.
There are still 39,000 requests blocked at the block rule, so it's working a bit but they are now getting past my challenge rules again - between 25% and 30% are anyway.
Numbers are still ok in the guest list at around 500 so it's not a major issue - until they swarm again perhaps, and at least 70% aren't solving the challenge. But the xxxers are scraping my site again.
I suppose at least it looks like my site is busy now
Claude has started to get bossy and argumentative
I think it's relevant. Just what are site owners supposed to do when Cloudflare can't stop residential proxies that solve challenges? While the numbers may be manageable now, it only takes a huge swarm of solvers to push numbers so high it takes your site off line. Not to mention the fact you can't stop them scraping your work.
However I like the idea of a honeypot, similar to Ozzie's registration spaminator, and I wonder if that could be achieved with Cloudflare snippets. ie something to click on that only bots see.
There are still 39,000 requests blocked at the block rule, so it's working a bit but they are now getting past my challenge rules again - between 25% and 30% are anyway.
Numbers are still ok in the guest list at around 500 so it's not a major issue - until they swarm again perhaps, and at least 70% aren't solving the challenge. But the xxxers are scraping my site again.
I suppose at least it looks like my site is busy now
Claude has started to get bossy and argumentative
I think it's relevant. Just what are site owners supposed to do when Cloudflare can't stop residential proxies that solve challenges? While the numbers may be manageable now, it only takes a huge swarm of solvers to push numbers so high it takes your site off line. Not to mention the fact you can't stop them scraping your work.
However I like the idea of a honeypot, similar to Ozzie's registration spaminator, and I wonder if that could be achieved with Cloudflare snippets. ie something to click on that only bots see.
Last edited:
My residential proxies are using Windows Chrome 151 as well and those are the ones that solve challenges. Some are Linux Chrome 151 but not that many.
They are probably not "using" these browsers but the bots that currently use these residential proxies hitting you identify themselves as these browsers. Small, but relevant difference.
