Which fix is that? Which addon?
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Human verification and added block country, I went to fighting this crap daily to get back to normal and actually better.
Could you explain please - how do you mean use with a blocked country (as per your review of the addon)?
Yeah, no. It's an add-on that lets bots hit the server and PHP interpreter (which powers XenForo). Not an ideal way to do this, especially for high-traffic forums. Very low traffic? Sure, maybe it will work...? I'm conscious of server loads, and keeping as much as possible away from XenForo is how most of us mitigate that.
The bot army needs to be properly mitigated before it hits our servers or, in layers, like Cloudflare -> server firewall and/or anubis -> web server config (htaccess or nginx equivalent) -> then XF to take care of what's left. Without upstream protection, bots hammering on an add-on like that would still bring a busy server to its knees.
Or for that matter, using htaccess to block some of this traffic is having Apache do all the heavy lifting (and again, the equivalent in nginx or other web server). It's still preferable than letting the bots through to XF though.
YMMV of course...
The bot army needs to be properly mitigated before it hits our servers or, in layers, like Cloudflare -> server firewall and/or anubis -> web server config (htaccess or nginx equivalent) -> then XF to take care of what's left. Without upstream protection, bots hammering on an add-on like that would still bring a busy server to its knees.
Or for that matter, using htaccess to block some of this traffic is having Apache do all the heavy lifting (and again, the equivalent in nginx or other web server). It's still preferable than letting the bots through to XF though.
YMMV of course...
Good point about .htaccess. I seem to have had a new issue today. My email is with Googleworkspace and had a few server errors today where it dropped some emails. Maybe because whats new is being hammered and triggering a lot of emails?
I don't think those are related--there's nothing in What's New that would trigger an email. It's just a search-generated page of the latest content.
Would Workspaces give you a warning if you sent too many emails over your quota? I've never hit it on ours, but we don't use it for sending forum mail.
The Block Country add-on blocks IPs from different countries ( I had issues with Russia and China)... The other add-on kept the bots away as they can't do the verification. It sounds too simple, but it's working!
Well as an experiment, I removed the referer from my URI path rule - meaning genuine guests will get challenged going to whats new, search or members as well. Just to see if the solve rate improved. It has made no difference at all, so as suspected the bots are just solving the challenge. 18% CSR rate so almost a fifth of them are getting past the rule. Also not happy to find they are scraping attachments - the site is image heavy.
Last edited:
So bot numbers in my guest list have been reasonable 350 or so the last day. But now they are scraping my articles (Bob's AMS) instead of "latest content" and I am really mad about that - the articles were a lot of work. I already have a challenge for that path and yet umpteen USA IP's are scraping them.
Update: it turned out it blocked people, despite the exceptions. They could click on "cookies" but as soon as they changed their preference, they were blocked out. I found that out myself when I got blocked out... So I deleted the rule. I have also edited my earlier post.
I had a similar thing! I turned my rate limiting rule back on (which previously had been doing virtually nothing as the bots changed IP's before they hit it), and added a second field - ams. This morning I got blocked "temporarily" hitting the rate limiting rule, switching between the home page and what's new - and a few members will have done this morning as well. It only lasted a few seconds but is pretty horrible! So taken that out of my rule again. It's supposed to be 10 hits in 10 seconds and I only did 2! But I switched between ams (home landing page) and then clicked whats new. So rate limiting wasn't quite working as it should have done. At least it messed up a few bots along the way but only slowed them down.
The numbers in my guest list are ok - it's what they are scraping that is the issue.
Incidentally - log analysis showed two scraper bots who, admittedly, declared themselves (but not that upfront if it took a log analysis to identify them) - which I have now blocked in .htaccess in my bad bots bit.
Reflectionbot
Lightpanda
Lightpanda is a headless browser which multiple companies can use and is freely advertised on its own website - it was scraping images. Reflectionbot is an AI bot.
I've turned on image hotlinking in Cloudflare now.
Just in case anyone wants to add those to a block list.
What does that do exactly?
I
"When another website displays an image using a direct link to your server, it is called hotlinking. Every time a visitor opens their page, your server has to spend CPU power and bandwidth to send them that image, essentially letting them steal your server resources and content for free.
By turning on Hotlink Protection in Cloudflare, the following happens:
From Google - who explains it better than I can. But I think it basically stops other sites re-using your images if they have scraped them.
"When another website displays an image using a direct link to your server, it is called hotlinking. Every time a visitor opens their page, your server has to spend CPU power and bandwidth to send them that image, essentially letting them steal your server resources and content for free.
By turning on Hotlink Protection in Cloudflare, the following happens:
- It checks the request source: When an image is requested, Cloudflare looks at the HTTP Referer header to see which website is asking for it.
- It blocks unauthorized sites: If the request is coming from your own domain (or search engines like Google), Cloudflare allows it. If it is coming from an outside domain trying to steal the image, Cloudflare blocks the request.
- It protects your bandwidth: The scraper or scraper-site will see a broken image icon (or a 403 Forbidden error) instead of your graphic, instantly saving your server from processing the load.
Hotlinking stops websites loading images from your website, onto their own, using your bandwidth.
I've been blocking hotlinking for at least 25 years now, using htaccess. It's one of those things we discovered in our logs--other sites using our bandwidth to display our images on their sites, which is wrong for all sorts of reasons. (It's like using an IMG tag where the source of the image is hosted on someone else's site.) It's nice having that option in CF with the flick of a switch, so we don't have to mess around with htaccess as much.
Back in those days, some of the more "adventurous" admins used to serve those sites a photo in response. So, rather than display the actual image linked to, htaccess could rewrite that to send an image of the admin's choice. Sometimes they were warnings (along the lines of "image linked without permission" or "stolen image!"), but the more adventurous of those occasionally got a bit vile.
Back in those days, some of the more "adventurous" admins used to serve those sites a photo in response. So, rather than display the actual image linked to, htaccess could rewrite that to send an image of the admin's choice. Sometimes they were warnings (along the lines of "image linked without permission" or "stolen image!"), but the more adventurous of those occasionally got a bit vile.
I'm glad you pointed that out.
On a possibly related note, or not, I have an issue where the main page of the XF site is displaying a blank page for some logged-out visitors. Yet if I add a nonsensical query string to the end of the site's URL, like
https://myforum.com?nonsense=1, it displays. It also works if they point to any of the forums individually. So I don't know what's going on just yet. I'm about to start disabling the rules in CF individually to see if they are affecting it. And I might clear the site's cache to see if that helps.And I'm still having issues with one of the CF rules where images in R2 buckets do not display. (They eventually start disappearing as the caches expire.) I probably have to whitelist something to get those to work, and I don't know if there is a header or cookie I need to whitelist also, but something is preventing them from displaying for some of our logged-in users.
On our busiest site though, I'm pleased with how the filters are working out. The last three swarms were handled with ease. I'm not worried about nitpicking the details unless this behavior changes.
This was our last bot swarm from yesterday evening. Notice how the origin (light blue plot) stays steady, without a single blip, while the mitigation is off the charts.
11:00am yesterday morning, only a very slight uptick as the swarm began:
And two nights ago, that swarm only resulted in a small surge in traffic, not even noticeable:
CF's filters are like...
This was our last bot swarm from yesterday evening. Notice how the origin (light blue plot) stays steady, without a single blip, while the mitigation is off the charts.
11:00am yesterday morning, only a very slight uptick as the swarm began:
And two nights ago, that swarm only resulted in a small surge in traffic, not even noticeable:
CF's filters are like...
