That's interesting...
I dealt with the search issue by disabling search for guests; they can go to Bing or Google. That means one less CF rule to deal with. Didn't realize they were also hammering /misc/cookies.
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
I dealt with the search issue by disabling search for guests; they can go to Bing or Google. That means one less CF rule to deal with. Didn't realize they were also hammering /misc/cookies.
/cookies has always been a top path on my forum. First path I put a managed challenge on. Now it's normal but the swarm that came in last night hammered it again.
I too had Search available for guests. Also had to disable it. A pity but man, did they hammer it! I have AndyB's Search Log add-on so I could see what guests were looking for. And then saw that these bots, with no access to member profiles were searching on member names within threads. It looked as though they were actually harvesting member data that way. So I immediately turned off Search for guests. That gave me a real scare as my first priority is the safety of my members. All the rest comes next.
Very interesting. That seems an odd thing for the AI bots to be training on. Unless they are somehow "threading" all of a user's posts together somehow. I've had guest viewing of member profiles turned off, but I am probably going to make a template edit so the usernames in the postbit do not display the "membercard." Or, aren't linked at all. I could probably do the same with member @ mentions also, as that really has no functional purpose except for logged-in members. I don't know if bots can access those or not (seeing that they're triggered by a tap or mouse click) but better safe than sorry. And like you say, I err towards member privacy over the guests on the Internet getting access.
I know some bots also follow the path to the style chooser; that's another feature I'm going to make a template edit for, so it will not appear unless a member is logged in. I doubt it will upset anyone who's a guest. And if they want that control, it's enticement for them to join.
I had thought of displaying on a first page of posts, but so many of our threads are dozens of pages deep and a lot of useful information is contained therein. So that isn't an option.
Overall I think that some of these features are better handled by making template edits to remove functionality for guests, rather than making rules. That means we have fewer holes we have to patch using CF's security rules.
When I get time tomorrow, I'm going to look into /misc/cookies, even if they're not hammering it now. That's why lurking here and absorbing these tips helps all of us in the long run.
That's what it looked like to me. The same user names were searched in different threads. Again and again. I too have guest viewing of member profiles turned off but there's still the user window. I have the location in the tooltip but I wish I could make that tooltip non accessible for guests. I looked but couldn't find a way.
Yeah? Not seeing that. Maybe 2000 requests in 24 hours or so.
But isn't Cloudflare the better option? Everything stopped there at least won't reach the server.
They're still targeting it. 13k hits in the last hour.When I get time tomorrow, I'm going to look into /misc/cookies, even if they're not hammering it now. That's why lurking here and absorbing these tips helps all of us in the long run.
I might be wrong but they hit the cookie banner to get rid of it, so they can "see" the full page clearly to index? They probably don't get a referer cookie, being bots, so get the cookie banner on every page they want to scrape and have to get rid of it.
I don't think so. A bot doesn't see a page like a human being does. It sees the source code, the html.
I had managed challenge on one rule for a while. I just changed it to interactive to see if it stopped more bots - which it did slightly. Regular users and members wouldn't have been challenged in that rule so it didn't matter.
But I kind of like the idea of an interactive challenge with the tick box in some ways - if a challenge has to be there at all - just from my experience of landing on other websites. I would rather tick a box and be done with it, than sit there with a page with a spinning circle, not knowing what is going on or how long it will take. So does managed challenge sometimes show nothing at all? Rather than the spinning circle page?
When I find out something, I can let you know. I've been too busy with multiple sites, dealing with all the bot traffic. Two of the forums are also long overdue for an upgrade to 2.3, and I haven't been able to put aside the time to do it thanks to all this messing around.
I don't see many accesses, certainly not enough to worry about, but I know some bots will wander in there.
True, it's better to stop everything long before it hits the server, and that is my preference. But at least for those who aren't stopped by Cloudflare, they can try to access search but won't be able to find it, so that's less of a load for serving a dead end vs. letting them get in and actually trigger the search. I'm pretty sure the What's New tab is powered by search, and it's so tempting to ditch that also...but I know it's also a very popular feature, and I'm sure the CF verified bots know to look there for the latest content to index.
Oof. I don't know what's up but today we've gotten slammed twice, and this is after my latest round of rules tweaking. I'm so over this.
Yet if we look at the bigger picture...like early this morning, when we had yet another round of bots hit us. In the course of an hour, there were 957k requests. 487k were mitigated by Cloudflare. 392k were served by our Cloudflare cache. 79k were served by origin (our server). So of all that traffic, our server only saw 8.2% of it. And we still got slowed down for about 5 minutes. Not even that severely, to be honest. One interesting spot I saw was around 7:09am, when our origin traffic dropped back to normal, as did the Cloudflare cache...but the mitigation continued for another nine minutes. This is part of the graph for the hour.
Normally, all three plots in the graph are concurrent, but this shows that CF was kickin' butt and takin' names as our origin traffic eased back to normal.
Tomorrow it's back to slaving over the rules yet again, as I'm now getting a warning from Google that they're seeing a lot of "403 Forbidden" results due to blocked access...and I have Google whitelisted, so no idea what's going on there.
As they say...back to the drawing board.
Why turn off guest searching for that? Put a rule on search. Every action you take because of bots that negatively affects people using your forum, just turns people away from your community, elsewhere, bots win.
Yeah, I've been thinking about it. I panicked when I saw what was happening and immediately disabled guest search in the ACP. Later as I became more familiar with Cloudflare I created a managed challenge for paths I couldn't block and added /search to it. So yeah, I could enable it again.
Well I've set up Precursor. Apparently I can set up rules on the free plan - 3 showing. So I've set a precursor rule for the whats-new path, but it's limited without an enterprise plan. With an enterprise plan you can set up a corresponding security rule that matches the precursor rule but sets a bot score - if the bot score goes below the number you set, it gets blocked - if you set the rule to block. I think that's how it works.
However as I don't have an enterprise plan I couldn't set up a security rule with a bot score. But the precursor rule is doing it's own thing and analytics/events show it is serving an interstitial page to every bot going through whats new. I'm not quite sure how that differs from the bots already being challenged in my security rule for whats new, except that precursor is also monitoring the behaviour with telemetry. Presumably it can see the bot is solving the challenge, and establish that it is a bot - not sure what it does with that information. It hasn't changed any solve rates so far anyway.
However as I don't have an enterprise plan I couldn't set up a security rule with a bot score. But the precursor rule is doing it's own thing and analytics/events show it is serving an interstitial page to every bot going through whats new. I'm not quite sure how that differs from the bots already being challenged in my security rule for whats new, except that precursor is also monitoring the behaviour with telemetry. Presumably it can see the bot is solving the challenge, and establish that it is a bot - not sure what it does with that information. It hasn't changed any solve rates so far anyway.
I don't get rounds - I get 24/7 hammering! Although admittedly the hammering is a bit less during the day than at night. It's quite satisfying to see how many Cloudflare prevented getting to the site when there are 500,000 or so IP's - but it would be nicer if more of them didn't get through.
Labyrinth does a good job at that, diverting bots into never ending rubbish pages of AI content. I sometimes search out how many are in it, and it does a good job diverting bots into it.
Yer... the non-interactive does have an issue with catching possible legitimate users. Just another tool with unintended consequences, I guess.
Yer... the non-interactive does have an issue with catching possible legitimate users. Just another tool with unintended consequences, I guess.
To my understanding, Precursor isn't even (directly) about prevention but about detection. It does some extra steps to get a more comprehensive understanding if a visitor is a bot or not, in the form of a score. Then that score can be used elsewhere in other rules, as a degree of confidence for bot vs human. But what do I know.
When my forum was hit on the /misc/cookies, it wasn't the frequency on the hits on the URL that was interesting, but the GET parameters that were passed in the URL. The requests seemed to use the page as some sort of a "safe target" for testing out brute force combinations of all sorts of GET parameters, not sure if random or scraped from elsewhere or generated. If they'd test it against an explicit login page, that would obviously stand out and possibly trigger protective measures. But an inocuous page that theoretically anyone should be able to visit and read the cookie policy? No idea what was up with that, other than this resembling very much a vulnerability scanner/penetration test for XenForo. I just found it weird and illogical for a bot harvesting contents, so I added it to the list of challenged URLs for bots.
In theory, CloudFlare started as a filter at the transport layer of networking, dealing with IPs and sources. But it blurs the lines by also having some degree of filtering at the application layer, as a man-in-the-middle between the origin server and the client, and with full access to the unencrypted traffic (HTTP requests and page responses). It is capable of analyzing HTTP requests before they hit the origin server and its own application-level rules (group permissions etc.), and injecting extra code in the pages being sent to the client. This is limited, however; CloudFlare doesn't have full access to the server session, to the user's role and limitations that XenForo can define and enforce, in order to block a request based on those criteria through CF rules before it reaches the server.
The point to this is that having CF rules DOES NOT exclude the need to make other necessary changes to usergroup permissions in the forum configuration, which is the only place where you can fine tune those. Sure, CF can act as a blunt tool and shield most of the undesired traffic, but the rest of the filtering has to be put in the forum.
A good example is the forum search: CF is the first defensive layer that can defend against most bots that incessantly hit directly the search feature without even having a server session or a cookie from a previous visit, and avoid even generating a server response for such a request. But for those visitors that pass through, it's best to fine tune the permissions on the forum itself: consider the search feature as a costly function of the server and therefore a privileged feature for members only. So use the forum as the second defensive layer to disable forum searching for guests group, to encourage them to register.
Layers.
When my forum was hit on the /misc/cookies, it wasn't the frequency on the hits on the URL that was interesting, but the GET parameters that were passed in the URL. The requests seemed to use the page as some sort of a "safe target" for testing out brute force combinations of all sorts of GET parameters, not sure if random or scraped from elsewhere or generated. If they'd test it against an explicit login page, that would obviously stand out and possibly trigger protective measures. But an inocuous page that theoretically anyone should be able to visit and read the cookie policy? No idea what was up with that, other than this resembling very much a vulnerability scanner/penetration test for XenForo. I just found it weird and illogical for a bot harvesting contents, so I added it to the list of challenged URLs for bots.
In theory, CloudFlare started as a filter at the transport layer of networking, dealing with IPs and sources. But it blurs the lines by also having some degree of filtering at the application layer, as a man-in-the-middle between the origin server and the client, and with full access to the unencrypted traffic (HTTP requests and page responses). It is capable of analyzing HTTP requests before they hit the origin server and its own application-level rules (group permissions etc.), and injecting extra code in the pages being sent to the client. This is limited, however; CloudFlare doesn't have full access to the server session, to the user's role and limitations that XenForo can define and enforce, in order to block a request based on those criteria through CF rules before it reaches the server.
The point to this is that having CF rules DOES NOT exclude the need to make other necessary changes to usergroup permissions in the forum configuration, which is the only place where you can fine tune those. Sure, CF can act as a blunt tool and shield most of the undesired traffic, but the rest of the filtering has to be put in the forum.
A good example is the forum search: CF is the first defensive layer that can defend against most bots that incessantly hit directly the search feature without even having a server session or a cookie from a previous visit, and avoid even generating a server response for such a request. But for those visitors that pass through, it's best to fine tune the permissions on the forum itself: consider the search feature as a costly function of the server and therefore a privileged feature for members only. So use the forum as the second defensive layer to disable forum searching for guests group, to encourage them to register.
Layers.
Yes I think precursor does give bot scores - the trouble is you can’t do anything with them without paying $5000 a month it seems! Enterprise plan. On that plan you can then say any bot score under 20 or whatever - block - and let everyone else through to a challenge or even no challenge. Not quite sure about that.
Hopefully it’s doing something even if it’s just teaching Cloudflare engines which are bots and hopefully it’s preventing them doing do much scraping. But it’s not actually removing or blocking them. I’m kind of thinking these sophisticated residential proxies (the ones that can solve challenges) might be too clever for precursor though as they change IP’s so rapidly.
As for labyrinth it would be nice to have more statistics to see what it’s actually doing - I did download an events log but all that seems to show is a list of IP’s and not how many actually followed the labyrinth links.
Anyway I’m spending too much time on all this and want to have a life! So I’m leaving as is for now. Unless the bot numbers go right up again. Rules are working at the moment. Members aren’t getting challenged - unless a genuine visitor happens to come from outside Uk, US, Canada or Europe - and there are some but not a massive amount to have to tick a box.
Hopefully it’s doing something even if it’s just teaching Cloudflare engines which are bots and hopefully it’s preventing them doing do much scraping. But it’s not actually removing or blocking them. I’m kind of thinking these sophisticated residential proxies (the ones that can solve challenges) might be too clever for precursor though as they change IP’s so rapidly.
As for labyrinth it would be nice to have more statistics to see what it’s actually doing - I did download an events log but all that seems to show is a list of IP’s and not how many actually followed the labyrinth links.
Anyway I’m spending too much time on all this and want to have a life! So I’m leaving as is for now. Unless the bot numbers go right up again. Rules are working at the moment. Members aren’t getting challenged - unless a genuine visitor happens to come from outside Uk, US, Canada or Europe - and there are some but not a massive amount to have to tick a box.
Last edited:
English: I received an AdSense traffic warning specifically because of these bots. I do my best to delete them whenever I can. They tend to target the "new messages" tab in particular. I will be taking stricter measures in the future; otherwise, the AdSense warning won't go away.
Last edited:
This is an UK-based international forum with English being the commonly used language to ensure that communication across language borders is possible. If you want that anyone is able to read and understand what you are saying you should post in English too, as everyone else does it.
Translation:
I received an AdSense traffic warning specifically because of these bots. I am doing my best to block bots from accessing the site. They are particularly targeting the "new messages" tab, so I am going to implement stricter measures there. Otherwise, the AdSense warning won't go away.
I feel better leaving it off, vs. hoping a CF rule might catch it. (I still can't trust CF rules 100%.) Visitors can use Google if they want to search the forums. When I do our upgrade, Google search will be in the search drop-down, so that will be available to everyone. Nobody has complained (and with this forum, they'll complain to the owner immediately if they don't like something), so I'm not worried about it.
