Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
You need to calm and chill about this. First, get rid of any rules blocking ASN's and IP's, they aren't effective in any way, shape or form.
Ensure AI Labyrinth and Bot fight mode are activated.
One rule to eliminate fake browsers.
Code:
(((http.user_agent contains "Chrome/1" and not any(lower(http.request.headers.names[*])[*] == "sec-ch-ua"))) or (((http.user_agent contains "Mozilla/5.0" and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")) and not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode"))) or (((http.user_agent contains "Mozilla/5.0" and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")) and not any(lower(http.request.headers.names[*])[*] == "accept-language")))) and (not cf.client.bot) and (not http.cookie contains "xf_user")One rule to reduce the residential proxies.
Code:
(http.user_agent contains "Chrome/120." or http.user_agent contains "Chrome/145." or http.user_agent contains "Chrome/133." or http.user_agent contains "Chrome/146." or http.user_agent contains "Firefox/135." or http.user_agent contains "Firefox/133." or http.user_agent contains "Chrome/131." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/136." or http.user_agent contains "Chrome/99." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/142." or http.user_agent contains "Chrome/65." or http.user_agent contains "Firefox/147." or http.user_agent contains "Firefox/144." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/117." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/106.") and http.referer eq "" and (not cf.client.bot) and (not http.cookie contains "xf_user")One rule to protect search.
Code:
(starts_with(http.request.uri.path, "/whats-new") or starts_with(http.request.uri.path, "/search"))
and not cf.client.bot
and not http.cookie contains "xf_user="One rule for the noise, put it last.
Code:
((http.request.uri.path contains "xmlrpc.php" or http.request.uri.path contains "/wp-login.php" or http.request.uri.path contains "/wp-admin/" or http.request.uri.path contains "/wp-includes/" or http.request.uri.path contains "/wp-content/" or http.request.uri.path contains "wlwmanifest.xml" or http.request.uri.path contains "wp-config.php" or http.request.uri.path contains "/.env" or http.request.uri.path contains "/.git" or http.request.uri.path contains "/vendor/phpunit" or http.request.uri.path contains "/cgi-bin/" or http.request.uri.path contains "/etc/passwd" or http.request.uri.path contains "eval-stdin.php" or http.request.uri.path contains "/.aws/") or (http.request.uri.path in {"/wp-json" "/wordpress/" "/wp/"})) or ((http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot" or http.user_agent contains "Google-InspectionTool" or http.user_agent contains "Mediapartners-Google" or http.user_agent contains "AdsBot-Google" or http.user_agent contains "YandexBot" or http.user_agent contains "Baiduspider") and (not cf.client.bot))Then one rule to manage AI, SEO crawlers, more noise.
Code:
(http.user_agent contains "meta-externalagent" or http.user_agent contains "ClaudeBot" or http.user_agent contains "Amazonbot" or http.user_agent contains "SemrushBot" or http.user_agent contains "DotBot" or http.user_agent contains "AhrefsBot" or http.user_agent contains "Bytespider" or http.user_agent contains "GPTBot" or http.user_agent contains "MJ12bot" or http.user_agent contains "AcademicBot" or http.user_agent contains "ExaSearchBot" or http.user_agent contains "ShapBot" or http.user_agent contains "PetalBot" or http.user_agent contains "meta-webindexer" or http.user_agent contains "ReflectionBot" or http.user_agent contains "yacybot" or http.user_agent contains "LinkupBot" or http.user_agent contains "compatible; crawler)" or http.user_agent contains "Scrapy" or http.user_agent contains "Sogou" or http.user_agent contains "Qwantbot" or http.user_agent contains "Barkrowler" or http.user_agent contains "curl" or http.user_agent contains "ClueWeb-Crawler" or http.user_agent contains "Turnitin" or http.user_agent contains "MathPicDatasetCrawler" or http.user_agent contains "PurityBot" or http.user_agent contains "img2dataset" or http.user_agent contains "quillbot" or http.user_agent contains "IbouBot" or http.user_agent contains "Go-http-client" or http.user_agent contains "Brightbot" or http.user_agent contains "Mediatoolkitbot" or http.user_agent contains "webzio" or http.user_agent contains "YisouSpider" or http.user_agent contains "AwarioBot" or http.user_agent contains "CCBot" or http.user_agent contains "okhttp" or http.user_agent contains "serpstat" or http.user_agent contains "DataForSeo" or http.user_agent contains "Google-Extended" or http.user_agent contains "Timpibot" or http.user_agent contains "cohere-ai" or http.user_agent contains "anthropic-ai" or http.user_agent contains "ZoominfoBot" or http.user_agent contains "Applebot-Extended" or http.user_agent contains "Diffbot" or http.user_agent contains "ImagesiftBot" or http.user_agent contains "FacebookBot" or http.user_agent contains "omgili" or http.user_agent contains "BLEXBot" or http.user_agent contains "W3C_Validator") and (not cf.client.bot) and (not http.cookie contains "xf_user")If there is anything left actually hitting your server after that, you now have a very limited access log to analyse, find the exact issue and the best method to add that into an existing rule.
Put them all on managed challenge, or whatever challenge you prefer.
Beg to differ on that... ASN to IP CIDR blocking should be reserved for the most abusive networks. It has legitimate use, but shouldn't be the be-all-end-all solution. Networks such as Huawei Cloud, TenCent Cloud (Aceville), Alibaba Cloud, among some others, all are deserving candidates for that particular use.
And sure, by banning various ASNs announced IP CIDR ranges might result in the "sh*t hitting the fan, and being spread out on all the walls" scenario (rather direct access, uses residential proxies), it does stop particularly abusive and toxic networks from being able to communicate with your servers from the get go. Effectively stunting their attempts, slowing down (or flat out stopping) their scraping abuse.
- https://www.spamhaus.org/drop/asndrop.json Spamhaus's list of abusive ASNs.
- https://github.com/bountyyfi/bad-asn-list - seems to interface with Cloudflare workers system for ASN management.
- https://www.webhostingtalk.com/showthread.php?t=1947073
- (comments link to WHT link above)
Agreed. I don't use it, as there is nothing that abusive that isn't being caught in my existing rules.
Once more, as said before: This is completely wrong and really bad advice.
Once more: That depends from your goal. As you wrote earlier in this thread: Your goal is exclusively to keep the server load load down and you don't mind at all about content being scraped. Other people may have different goals.
But even with just server load in mind: Country, ASN and IP blocking is simple and effective and very cheap. You just have to care about possible collateral damage through accidental overblocking. In practice this is not a real issue in most cases as long as you only block server ranges - typically there is barely any legitimate purpose why servers should send any request to your forum aside of well known search engines, the rare occurrence of legitimate proxies or legitimate VPN end points and possibly the also rare occurrence of embedding /prerendering through i.e. Mastodon instances. Other than that it is straight forward, a simple "yes" or "no" while your ruleset may or may not work and while it works today it may be circumvented tomorrow. So it makes absolute sense to hard block areas of the net that you have no interest in or that are bad to the bone and filtering the rest more intelligently. This way you save on precious resources and gain more safety on top. Why would you even bother to have a closer look at any traffic coming i.e. from
AS132203 Shenzhen Tencent Computer Systems Company Limited
or many, many other ASNs? It is 100% crap anyway. Just ground it as the request arrives and you are done.
Of course, if you delegate all the filtering to Cloudflare you don't have to care about resources needed for filtering there - however, not everyone wants to use Cloudflare and even with them the ruleset is prone to be tricked out while simple straight blocking is not.
Intersting. Of all those ASNs I do have only a few in my list (but quite a bunch of countries blocked, so possibly caught a relevant of those ASNs automatically as well)
This seems a very interesting list and pretty large as well:
Possibly worth having a closer look.
Last edited:
This is where Cloudflare's custom list feature under the premium account would make rules covering thousands of ASNs so much more elegant and simple...
I too join the chorus that the ability to block entire subnets and ASNs is a mighty tool that has its place in our toolbox. Ignoring it is a choice, and ideally a well informed one. There is no right or wrong, it's just whatever works for you (which might not work so well for others).
But I would also appreciate seeing webhosts enhancing their management portals for clients with similar filtering controls for their network equipment - to give us the ability to add our own firewall filters without having to pass the traffic through a third party service. Software Defined Networking has been a thing for years. There are APIs and automated tools that detect intrusion attempts or unusual traffic or periodically checking RBLs. The hosters themselves most likely do not SSH into every hardware firewall or router to manage rules through the CLI, but through centralized management portals for their entire fleet of network devices. So the technical capability exists, we only need the software that gives webhosting clients a limited access to some operations.
I too join the chorus that the ability to block entire subnets and ASNs is a mighty tool that has its place in our toolbox. Ignoring it is a choice, and ideally a well informed one. There is no right or wrong, it's just whatever works for you (which might not work so well for others).
But I would also appreciate seeing webhosts enhancing their management portals for clients with similar filtering controls for their network equipment - to give us the ability to add our own firewall filters without having to pass the traffic through a third party service. Software Defined Networking has been a thing for years. There are APIs and automated tools that detect intrusion attempts or unusual traffic or periodically checking RBLs. The hosters themselves most likely do not SSH into every hardware firewall or router to manage rules through the CLI, but through centralized management portals for their entire fleet of network devices. So the technical capability exists, we only need the software that gives webhosting clients a limited access to some operations.
Well I had to ramp up the challenge on my last rule last night so most visitors would get a challenge (although still excluded Uk and US). Numbers started to fall by the minute and are right down this afternoon, so I decided to put everything back to where it was before I enabled AI crawl control, which seemed to trigger the bigger residential proxy swarm - although that could have been coincidence. And whitelisted Europe again in my country challenge.
See what happens now everything is back to where it was yesterday.
See what happens now everything is back to where it was yesterday.
Last edited:
Interesting discovery. It seems my limited guest viewing addon is why Claude was hitting registration over and over and got caught in a loop. Because I'm using an older version from before AI crawlers existed. The older version just whitelists SEO crawlers. Unfortunately the addon does nothing to stop residential proxies as they change their IP's so much they look like a new guest all the time so never get hit with the registration page.
So I could potentially harden the limited guest viewing so all guests are asked to register as soon as they hit the site - ie no page views or warnings (with a note explaining it's to prevent malicious bots) and bots wouldn't get past the 0 views registration page. But SEO bots would still get through. However the bad bots would probably then try and register and could affect my email account (Google Workspace) with too many bounced emails .............
So I could potentially harden the limited guest viewing so all guests are asked to register as soon as they hit the site - ie no page views or warnings (with a note explaining it's to prevent malicious bots) and bots wouldn't get past the 0 views registration page. But SEO bots would still get through. However the bad bots would probably then try and register and could affect my email account (Google Workspace) with too many bounced emails .............
Last edited:
Don't forget to lock down the forum permissions for Guests group. I just took a look - search is disabled an access to member profiles is prohibited, but I can still see who's currently online, or that big block of "Members recently online". Do guests really NEED to see that and trigger another database query for it? And many details of a member's profile are a mouse hover away in that big list... Let people outside the garden peek through holes in the fence, but keep some of the goodies away from prying eyes.
Disable member tooltip for guests:
Disable member tooltip for guests:
![[OzzModz] Disable Member Tooltip](/community/data/resource_icons/8/8541.jpg?1634773027){kind=icon}
Last edited:
This is really interesting--having CF workers access other resources, like a list of ASNs, and being able to add onto our filtering.
All good points. I've actually done a few of these since the days of vBulletin--I think I either changed permissions or modified the actual files to get rid of the Members Online block and a few other things way back then.Don't forget to lock down the forum permissions for Guests group. I just took a look - search is disabled an access to member profiles is prohibited, but I can still see who's currently online, or that big block of "Members recently online". Do guests really NEED to see that and trigger another database query for it? And many details of a member's profile are a mouse hover away in that big list... Let people outside the garden peek through holes in the fence, but keep some of the goodies away from prying eyes.
Disable member tooltip for guests:
That tooltip removal add-on only works on version 2.2, but it wouldn't take much to look at the template edits and change it for 2.3 and beyond. I agree with the "peeking through holes in the fence" idea. Preventing guests from using the features is an enticement to get them to join.
The US is one of the largest hosts of bots... you will continue to get smashed if you aren't actioning headers and such methods against US traffic.
The harder you make it to view your site, the less people will register or interact with it. That is decades old human psychology assessment on webpage interaction.If your language is left to right, you browse a webpage the same. If right to left language, browse the site that way. Your eyes move like that due to your trained reading. This is all marketing 101 stuff which you need to understand to run a social interaction website, let alone a website where you keep the visitor engaged to improve your conversion, whatever that is.
I am but this is just in a final "catch leftovers" rule
Maybe you should try to understand it yourself first before giving advice - it could avoid giving wrong advice. Proof: I limited guest views to the first post of a thread and locked down some forums to just the thread titles, articles to just the first couple of lines and hid a bunch of other stuff. This along with a note explaining why.
Consequence: My registrations have solidly tripled since then. The exact opposite of what you claim would happen.
The trick is to give proof that it is interesting and valuable to read the site (therefor the first post is visible) but if you want to see more people have to register - and this is what they do. As a consequence I gained more than 2000 new registrations since February, most of them are just lurkers (as to be expected).
BTW: Facbook, Instagram and loads of other very successful platforms harshly limit content browsing for guests and have done so already years ago. According to you they all have absolutely no clue how to run an internet platform successfully. I heavily doubt that you are right with your claims, based on evidence.
On our busiest forums, we have nearly 25 years of information that is buried deep in nearly one million threads and over 38 million posts; threads can run dozens of pages long. The Googles and Bings of the world index that and it sends us traffic.
So it depends on the forum if limited content for guests will work for them or not. For ours it won't. If I had more proprietary information, where indexing didn't matter as much. I would lock them down.
So it depends on the forum if limited content for guests will work for them or not. For ours it won't. If I had more proprietary information, where indexing didn't matter as much. I would lock them down.
This limited guest addon doesn't stop search engines - they are allowed through. So it's just guests who aren't search engines. Even so, being asked to register before they have even seen the site, could be offputting. I've left it as it is for now with a few views first.
I'm not sure it would work to stop bots that go direct to a path anyway. Or the bots would start to fake being search engines or something.
My forum has also been a very prolific space for computer nerds. We took pride in discussing the finest intricacies of modding, cooling, overclocking and such. We had threads going back 20+ years with a wealth of knowledge.
Until one day I realized that most of that information is technologically obsolete. Who will care about contents discussing AMD K6-II processors which went to recycling more than 10 years ago, and those still found in the wild are either in weird museums of junk, or withhoarders collectors, but not in actual use? Also, many, many, many of those threads stopped being active years ago. So it was time to prune them, tag them as Archived, and move them out of normal view.
(Same happened on a second forum focusing on some sports niche - Invision Power Board actually had a feature to move inactive threads into another database table for static contents, to speed up queries for new stuff or indexing of the now much smaller table for active threads.)
The point is, the discussion about content relevance over long periods of time, for search indexing and attracting new visitors, is heavily dependant on the general area of discussion. Gardening or pet care, sure, that won't be much different 25 years later. Hot deals on Black Friday in the Sears catalog, not so much.
So all rules and decisions can have a more or less limited reapplicability. Each can learn from others' experiences, but has to decide or test what works for them.
Until one day I realized that most of that information is technologically obsolete. Who will care about contents discussing AMD K6-II processors which went to recycling more than 10 years ago, and those still found in the wild are either in weird museums of junk, or with
(Same happened on a second forum focusing on some sports niche - Invision Power Board actually had a feature to move inactive threads into another database table for static contents, to speed up queries for new stuff or indexing of the now much smaller table for active threads.)
The point is, the discussion about content relevance over long periods of time, for search indexing and attracting new visitors, is heavily dependant on the general area of discussion. Gardening or pet care, sure, that won't be much different 25 years later. Hot deals on Black Friday in the Sears catalog, not so much.
So all rules and decisions can have a more or less limited reapplicability. Each can learn from others' experiences, but has to decide or test what works for them.
Last edited:
It could be quite enlightening if you could simply tell the name of the add on. Also, regarding "doesn't stop search engines" it is the question how it does that: If it simply refers to the user agent it is easy for bots to fake being a search engine bot. I.e. yesterday I had a series of requests on one of my test sites scanning for vulnerabilities and claiming to be Googlebot - which it wasn't:
Code:
94.154.46.243 - - [23/Sep/2026:03:04:37 +0200] "GET /wp-config.php.save HTTP/1.1" 444 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"In fact it came from a well known dark source:
So the add-on would need more sophisticated ways for detecting search engines such as reverse DNS and checking against the published ip-ranges of search engines. Does "your" add on do that?
What exactly are "search engines"? Indexing bots? Of which search engines? There are a plethora aside of Google and Bing. Which do you refer to?
Locking guests out completely with absolutely nothing of the content visible w/o being logged in is probably not a good idea in terms of registration. Also, as others say, it really depends from the character, content and business model of the forum. If you depend from ads locking out guests is probably not a good idea and also not, if you don't have exclusive content that cannot be found anywhere else. Users tend to go the easiest way and registration is a hurdle. If they can get what they are looking for w/o registration on another site or forum most won't register on your site.
Well 3 views then being asked to register has worked well over the years. I don't know how it allows search engines through, or which ones, that will be in the developer's code, which gets updated now and then. I assume it allows crawlers that declare themselves as well and then up to you if you want to block them, as I have Chat GPT user appearing. Although noticeably it was blocking all versions of Claude until I updated the addon, hence Claude hitting registration over and over as it hit a wall. Since I stopped blocking Claude, it hasn't reappeared in any form, although Anthropic appeared the odd time.
It's Wutime's limited guest viewing addon - been using it for years. Result being, I get a lot of registrations - because they want to read the articles. Some will go on to post from that as once they register, they get an automatic welcome DM. Others just read and go. But that's ok. Some come back later and ask questions. So plenty of members, but just a hardcore of maybe 20 or so who post fairly regularly.
Incidentally the solve rate on my Cloudflare path challenge has got worse and worse. It's now 26%. Numbers on the site have stayed low the last 24 hours, but if they swarm again 26% can solve the challenges.
It's Wutime's limited guest viewing addon - been using it for years. Result being, I get a lot of registrations - because they want to read the articles. Some will go on to post from that as once they register, they get an automatic welcome DM. Others just read and go. But that's ok. Some come back later and ask questions. So plenty of members, but just a hardcore of maybe 20 or so who post fairly regularly.
Incidentally the solve rate on my Cloudflare path challenge has got worse and worse. It's now 26%. Numbers on the site have stayed low the last 24 hours, but if they swarm again 26% can solve the challenges.
