Yes maximum 4096 characters - that rule above I pasted has about 1750. You can add more for just an expression than if you use the field boxes I think. Using the field boxes only seems to stop the character count early, but using expression allows the 4096
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
I wonder if any of this relates to why my forum loads OK, but attachments, avatars, and other things are broken when served from an R2 bucket. I can't turn off hotlinking completely, but I may turn it off in CF and go back to doing it in htaccess where I can control which domains can access it. I may, however, disable it temporarily to see if that is the issue. My R2 bucket's public domain (avatars, etc.) is a subdomain of our forum domain.
As an aside:
Why not use
not http.cookie contains "xf_user" ? Or am I missing something? My very first rule is a whitelist aka "skip" rule and this is right at the top of it, along with any IP addresses I use.
This. Exactly.
AI scrapers don't have my permission, nor have they paid, to use our content to generate their income and please their shareholders. So it's two tasks for me--filter out the bad bots and AI scrapers to limit our information as much as possible, and to chase away the bots that would drag our traffic down.
@Anthony Parsons , a quick question regarding your two filter examples. This one was from a week or two ago:
And this one from Monday:
The earlier rule seems more concise, and both seem to handle browser impersonation. Or do you use both?
Also, could I shorten a block of code like this:
...to this?
(It should also cover any of the versions in between 100 and 139, for the older browser versions. Or is it better to have the absolute versions listed?)
Gonna say... pretty happy with my current ruleset. My traffic is now normal, 165k uniques, around that 150k normal daily, and WAF events are doing their job, getting rid of 130k of cf.client.bot useless crawlers from my site. That browser impersonator... still the best rule to date, it has 9 solves from 280k events... that is the single most effective rule I have ever employed to my knowledge.
My current browser impersonator rule:
My current browser impersonator rule:
Code:
(
(
(
http.user_agent contains "Chrome/1"
and not any(lower(http.request.headers.names[*])[*] == "sec-ch-ua")
)
or...
Code:
(
(
(
http.user_agent contains "Chrome/1"
and not any(lower(http.request.headers.names[*])[*] == "sec-ch-ua")
)
or
(
http.user_agent contains "Mozilla/5.0"
and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
and not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
)
or
(
http.user_agent contains "Mozilla/5.0"
and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
and not any(lower(http.request.headers.names[*])[*] == "accept-language")
)
or
(
http.user_agent contains "Mozilla/5.0"
and http.user_agent contains "Version/"
and http.user_agent contains "Safari/"
and not (http.user_agent contains "Chrome/")
and not (http.user_agent contains "Chromium/")
and (
not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
or not any(lower(http.request.headers.names[*])[*] == "accept-language")
)
)
)
and not (
http.cookie contains "xf_user="
or http.cookie contains "xf_session="
)
)And this one from Monday:
Code:
(
(
http.user_agent contains "Chrome/65."
or http.user_agent contains "Chrome/99."
or http.user_agent contains "Chrome/100."
or http.user_agent contains "Chrome/101."
or http.user_agent contains "Chrome/103."
or http.user_agent contains "Chrome/104."
or http.user_agent contains "Chrome/105."
or http.user_agent contains "Chrome/106."
or http.user_agent contains "Chrome/107."
or http.user_agent contains "Chrome/108."
or http.user_agent contains "Chrome/109."
or http.user_agent contains "Chrome/110."
or http.user_agent contains "Chrome/111."
or http.user_agent contains "Chrome/112."
or http.user_agent contains "Chrome/116."
or http.user_agent contains "Chrome/117."
or http.user_agent contains "Chrome/119."
or http.user_agent contains "Chrome/120."
or http.user_agent contains "Chrome/123."
or http.user_agent contains "Chrome/124."
or http.user_agent contains "Chrome/131."
or http.user_agent contains "Chrome/133."
or http.user_agent contains "Chrome/136."
or http.user_agent contains "Chrome/142."
or http.user_agent contains "Chrome/145."
or http.user_agent contains "Chrome/146."
)
or
(
http.user_agent contains "Firefox/133."
or http.user_agent contains "Firefox/135."
)
or
(
http.user_agent contains "Safari/"
and not http.user_agent contains "Chrome/"
and not http.user_agent contains "CriOS/"
and not http.user_agent contains "FxiOS/"
and
(
http.user_agent contains "Version/13."
or http.user_agent contains "Version/14."
or http.user_agent contains "Version/15."
or http.user_agent contains "Version/16."
or http.user_agent contains "Version/17."
)
)
or
(
http.user_agent contains "CriOS/99."
or http.user_agent contains "CriOS/100."
or http.user_agent contains "CriOS/101."
or http.user_agent contains "CriOS/103."
or http.user_agent contains "CriOS/104."
or http.user_agent contains "CriOS/105."
or http.user_agent contains "CriOS/106."
or http.user_agent contains "CriOS/107."
or http.user_agent contains "CriOS/108."
or http.user_agent contains "CriOS/109."
or http.user_agent contains "CriOS/110."
or http.user_agent contains "CriOS/111."
or http.user_agent contains "CriOS/112."
or http.user_agent contains "CriOS/116."
or http.user_agent contains "CriOS/117."
or http.user_agent contains "CriOS/119."
or http.user_agent contains "CriOS/120."
or http.user_agent contains "CriOS/123."
or http.user_agent contains "CriOS/124."
or http.user_agent contains "CriOS/131."
or http.user_agent contains "CriOS/133."
or http.user_agent contains "CriOS/136."
or http.user_agent contains "CriOS/142."
or http.user_agent contains "CriOS/145."
or http.user_agent contains "CriOS/146."
)
or
(
http.user_agent contains "FxiOS/133."
or http.user_agent contains "FxiOS/135."
)
)
and http.referer eq ""
and not cf.client.bot
and not http.cookie contains "xf_user"The earlier rule seems more concise, and both seem to handle browser impersonation. Or do you use both?
Also, could I shorten a block of code like this:
Code:
http.user_agent contains "Chrome/65."
or http.user_agent contains "Chrome/99."
or http.user_agent contains "Chrome/100."
or http.user_agent contains "Chrome/101."
or http.user_agent contains "Chrome/103."
or http.user_agent contains "Chrome/104."
or http.user_agent contains "Chrome/105."
or http.user_agent contains "Chrome/106."
or http.user_agent contains "Chrome/107."
or http.user_agent contains "Chrome/108."
or http.user_agent contains "Chrome/109."
or http.user_agent contains "Chrome/110."
or http.user_agent contains "Chrome/111."
or http.user_agent contains "Chrome/112."
or http.user_agent contains "Chrome/116."
or http.user_agent contains "Chrome/117."
or http.user_agent contains "Chrome/119."
or http.user_agent contains "Chrome/120."
or http.user_agent contains "Chrome/123."
or http.user_agent contains "Chrome/124."
or http.user_agent contains "Chrome/131."
or http.user_agent contains "Chrome/133."
or http.user_agent contains "Chrome/136."
or http.user_agent contains "Chrome/142."
or http.user_agent contains "Chrome/145."
or http.user_agent contains "Chrome/146."...to this?
Code:
http.user_agent contains "Chrome/65."
or http.user_agent contains "Chrome/99."
or http.user_agent contains "Chrome/10"
or http.user_agent contains "Chrome/11"
or http.user_agent contains "Chrome/12"
or http.user_agent contains "Chrome/13"
or http.user_agent contains "Chrome/142."
or http.user_agent contains "Chrome/145."
or http.user_agent contains "Chrome/146."(It should also cover any of the versions in between 100 and 139, for the older browser versions. Or is it better to have the absolute versions listed?)
Tried before same issue, none of them work when it comes to the image-related problems I mentioned
I was asking more in terms of which CF expression would properly detect the xf_user cookie.Tried before same issue, none of them work when it comes to the image-related problems I mentioned
I've tried numerous things for broken R2 images myself and have gotten nowhere, and nobody seems to know what the issue is. I may just get rid of R2 buckets, as they seem to be more trouble than they are worth.
I don't use both, but I have.
The residential proxy rule I have posted here, that targets user agent versions which appeared consistently in my logs from residential proxies, ie. the pattern formed from them doing the same thing x times from rotating IP's.
The browser impersonator rule detects incomplete headers.
I use the browser impersonator only at present, because the two rules combined, the residential proxy really just picked up some stragglers and didn't help much. Yes they sort of capture most of a market, combined they do more, but if 200k daily (normal daily use), and the second is doing 30k daily, it translates to stuff all of a problem in my view. Swap them around and get similar results, but I found the browser impersonator did slightly better, so I just use that.
I have never been trying to stop everything nasty, in fact, I don't care that much about it, I just want to stop the worst offenders that cause server problems, and my rules do that. Facebook is actually one of the worst offenders, so if you allow cf.client.bots then you are putting one of the worst offenders for scraping onto your site. This is why I decided to tune what I really needed and block the rest of cf.client.bots, cause its just mostly garbage.
My real traffic, is about right below. If it sits around the 150k - 200k uniques, that is about correct. Vastly above that, there is a problem trying to get in, and from all of the attempts nowadays, my security rules below mimic any attempt to overwhelm (spike uniques matches spike WAF rule blocking it), so those four rules are doing their job, and the most important one is last, the load bearing one. Anything that gets past the browser impersonator and tries to bring the server down with search, is caught and stopped.
And the most important part, is that cache figure. 65% of requests are being served from CF cache and never touching the origin. I can go balls to the wall, add residential proxy rule and I would say... combined that takes 95% of the garbage from my server, which would improve further my caching output, but the server is highly tuned. That is where I spent my time, not chasing bots around. My caching is realistically higher than that figure, because what is hitting the server also gets cached data from Redis.
Block the worst offenders, tune your server to the enth degree, that should be every admins focus. I can have millions of uniques and not crash my server, providing they aren't all hitting search at once. I've had a million daily uniques with search open, and it survived just fine. About 1.5M daily uniques onwards, with search open, starts causing issues and nothing will solve it other than more cpu cores, so I protect search and honestly, the rest can have at it, because everything after the first pull, all further bots, will get primarily cached data, which is what you want. First one is load, everything after trying to kill your server, has little impact getting cached data. Search and DB are both fully in RAM too.
Chasing bots is not the solution. I analysed the logs, found the logs themselves weren't capturing the actual data I truly needed to get a true fingerprint of the issue, so I had to change what the logs captured first, that increased the log file size significantly, so I implemented compression, download and rotation policy on logs to my system, that data allowed me to fingerprint the true issue with the right header info, and hence the rules I have shared here. BUT... that is what hits my site, you may get slightly different different data from your logs, requiring a slightly different rule.
I don't use R2 on my forum, but I use it on other sites, never had an issue with it. I don't run an image heavy forum, mine is text heavy.
I'm the same way--I'm not out to get everything, but if the server is running well and my visitor numbers are around normal, it's not worth my time to find stragglers. We've had four quiet weeks in a row now (in terms of origin traffic), and our single large spike (largest ever) didn't affect origin traffic with my latest set of rules. I'm only looking at streamlining the rules I have so they are easier to manage.
I similarly whitelist only a small number of scrapers, and check that they are legit via cf.client.bot. All the rest get challenged.
I'm not sure if full-size attachments are affected (
internal_data), but avatars (from data) were disappearing, and that is where members have seen the issues. It'll take some detective work when I have the time.
I don't know how you're storing yours, or the setup, but my use of R2, I assigned different image tasks to different buckets, each having their own secure sub-domain via R2 handling, ie. avatar.domain.com blog.domain.com task.domain.com etc. But I also use an incoming bucket too... which processes the images into the actual used buckets, which is a bit special in its setup to ensure security and everything is sanitized correctly, nobody can do nasty things and all.
When I have time I will play around with XF and R2, see if I can find any issues with it.
I know that technically correct is the best kind of correct, however the point remains: whatever your goal is, if it is unrealistically high (like, perfection: 0 bots, 0 scrapers, 0 downtime, 0 latency), at some point beyond a "good enough" level it becomes ineffective to continue down this rabbit hole, to invest more and more effort in controlling the conditions which generate smaller and smaller gains towards that impossible goal.
One has to recognize when they're become obsessively attached to monitoring online status and logs and fine tuning rules and spending a couple free days fighting virtual bots instead of enjoying life offline. Remember that cartoon, "Honey, I can't come to bed right now, someone is wrong on the internet"? Yeah, that kind of obsession.
On Monday I spent a couple of hours fine tuning the rules which dropped incoming traffic from roughly 1500 per hour to about 300 - 80% decrease. I was ok with 1500 as well, because most of that traffic would hit the forum and discover that all content is behind a registration wall, so scraping failure rate is 100% anyway, but the effort to filter traffic upstream was not significant and the goal was completely achievable.
To get another level of 80% drop from those 300 to 60 it would probably take at least a few days of full time work, and there is a significant chance that I will not be able to reach that goal. Then to get again 80% down to 12 it will be many months, with an even slimmer chance of success. The cost increases exponentially, the actual benefits become negligible. There is a point where I absolutely must stop, otherwise this will consume my entire life and still not reach the ideal zero. And that's the point I was trying to make, not to dismiss the different goals of different people.
Look at the last 10-20 pages of this thread - it's very little new information or strategy for fighting bots, and almost all an endless optimization towards an impossible zero bots on a forum. One has to acknowledge the limitations of reality, find satisfaction in what is reasonably achievable, and move on.
Try this:
Code:
http.cookie contains "xf_user="For R2, are you using the CloudFlare plugin by any chance? To me, the configuration needed for it to work has not caused any issues for users, even with the filtering rules I added afterwards, and while not allowing direct access to the buckets' contents (all traffic is proxied through XenForo's engine and permissions and all that, none of the R2 is directly exposed).
This is known as Pareto principle: You reach 80% of the outcome with 20% of time while the last 20% need 80% of time. In your case, having all content behind the registration wall, it would be pointless to go down that route as long as the load on the forum is ok as your content is protected anyway. But on the other hand, having everything behind the registration wall leads to discoverability of your forum probably having fallen to close to zero. You made your decision, others may decide differently where they set their sweet spot. On my forum guests see the first post of a thread only and this only in selected forums while others let their content completely open for guests and some do not mind with their content ending up in AI models while others even hope it will.
So it is pretty obvious, that people have different goals and therefor end up with different strategies. All of them are equally valid and neither of us should pretend that the own strategy would be the only valid one. The sad point is that we have to deal with that mess at all. The consequence of the scrapers and bots is that the open internet where you could find and read information freely has come to an end due to those AI guys who even profit from that
Last edited:
I'm almost at the point of petitioning CloudFlare to offer a new filtering feature called „Ukrainian solution” 
If dropping big drones into Russian datacenters stopped the hate and disinformation flow from fake social media accounts in US MAGA, German AfD and a few others, surely a more resourceful corporation like CloudFlare could take a hint at more effective solutions against residential proxies, no? 
If dropping big drones into Russian datacenters stopped the hate and disinformation flow from fake social media accounts in US MAGA, German AfD and a few others, surely a more resourceful corporation like CloudFlare could take a hint at more effective solutions against residential proxies, no?
The only thing bothering me, is that the last time I remember checking the search protection, it had a consistent solve rate around 500 - 1000 per day. That is now 3k - 4k daily. For traffic that has not otherwise increased, that solve rate is significant. Not sure if its a good thing or bad, just yet. Likely bad... but I have my fingers crossed. Will check it in a week to see if some sneaky bot master is working on their solving solution for their next attack phase.
With Cloudflare being the by far most popular protection mechanism against bots it seems logical that the baddies focus their power on overcoming their captcha and it is a matter of time until most of them are able to, very much like it happened to other captchas in the past. Probably CF will sooner or later improve their Captcha but it is a rat race.
For the time being it may be an option to use a less popular solution, moving out of the main track of the scrapers. A little bit as in the past people using Macs had no issues with viruses and worms than Windows users, not so much b/c MacOS was such a barrier but mainly b/c Windows was way more popular and therefore a way more attractive target.
FYI I was laid up/unwell, so it was an extremely useful thing to do keep me mentally occupied/learn a bit, while unable to do much else, and also achieving something. It is insulting to call it an "obsession". Perhaps you have misunderstood the goal. It was analysing/monitoring after a major change and understanding what was happening where at that point - not chasing every last bot - but understanding which ones would get through challenges - because that helps have a plan ongoing.
I learned a lot in that couple of days - and in between that did various other interesting things as well thank you.
It was an analysis exercise after a big change. However, point taken there are more interesting things to do but each to his own.
Not obsession - I had an idea the other day, it worked and it killed 99.9% of them stone dead - meaning I didn't have to worry about these swarms shooting up every day again, or the frustration of Cloudflare challenges only being partially effective, and could get on with my life so to speak - so to me, that's win win.
That then created an interesting situation where there wasn't much left to look at so I learned a few things. All experience is useful. However yes I was picking a few things off and also sharing those results in case the info was helpful to anyone else.
I'm still gobsmacked that desktop whats new rule worked - and is still working - and if it stops working - well the other rules are probably enough.
I found it an incredibly useful couple of days - if 30% of bots can get past Cloudflare challenges, determining what they are/where they come from is very helpful knowledge.
This is not about perfection or it's opposite - complacency. It's about being in control of your own forum and website - and not letting some unwanted automated traffic control whether things get better or worse isn't that what everyone on here wants?
Just different views - yours is - hey its not crucial just ignore. Mine was - I have this time I'll see if I can solve something like a puzzle - and have some control over it. However I'm not up on coding and rule syntax, so asked someone to check one of my rules after adding something, to make sure the brackets were correct. I'd added @Anthony Parsons fake bots rule at the end of an existing block rule, then added a couple more bots and wasn't sure if I'd messed up the brackets to see if not cf.clientbot was actually applying to the right thing. It's on the previous page.
Last edited:
I'm very happy with the results. While I hadn't intended to stop all the AI crawler bots, I more or less have, with one rule. So not only are there minimal bots on my site, other than the ones I choose to have there, but they are not stealing my content and scraping my articles, and virtually no one from USA, Canada, Europe or Uk is being challenged - ie my real users.
Last edited:
What is the one rule to rule them all?
I've done all this, yet users report broken images. That exact cookie rule has been in my very first rule, whitelisted.
Only the attachments go through XF--the avatars and such (in the data directory) all use the R2 bucket's subdomain. I also have a worker set up to proxy image links pasted into the forum.
To add insult to injury, this morning's newest surprise:
1) No alerts from Cloudflare that they're having an issue; 2) nothing touched on Cloudflare in a couple of weeks; 3) nothing touched in the XF installation in at least a month. So this happened overnight.
So yeah...something else to troubleshoot. R2 buckets are just another failure point I don't need, so I'm most likely going to get rid of them. I'd rather pay for the storage at the host and not have a third party's failure make our site dysfunctional.
