Technically yes though you still have quota limits for WAF rules/rate limits at Enterprse plan level which must be shared with all sites that you extend SAAS feature for. On Enterpise plan WAF rules = 1000 rules and Advance rate limiting you get 100 rules, API JWT validation rules 100 rules, User Agent rules 1000 rules. For WAF rules for me I already using 225 rules!
Yer... I just built my own heuristics system that analyses monthly logs and finds patterns over consecutive months, then provides recommendations and API into CF to adjust rules quickly at a click. This whole addon thing, its a nightmare. One is crashing memory and cpu, then having to adjust how it operates to partially operate without doing one or both of those things. Another just quickly costs more and more via API checking, and the more you use it, the more bots hit your site to keep you moving up the payment plan. Bots have a behavioural pattern, so without affecting day to day running of the site, I will find those patterns over months of data, the more data the better the analysis, and then deploy targeted rules to counter the problem, not random IP/ASN/Country blocking that is happening.
Sorry, not yet. I am dabbling in another direction right now... the below is just one months data. No overhead this way. I implemented heuristic analysis of monthly access logs, aiming for superior detection accuracy through pattern-based traffic analysis rather than static rule matching. Going down the session fingerprinting route to see where that takes me for a robust recommended ruleset.
Nice. You might want to also sign up for a AbuseIPDB account too that has crowd sourced IP intelligence as well. I use it via custom CSF Firewall integration tool I developed at https://github.com/centminmod/centminmod-abuseipdb-reporter so CSF Firewall bad IPs get blocked at firewall level and also submitted to AbuseIPDB database. I now going to add it as 5th IP data intelligence provider for my Cloudflare aggregated analytics script just to see what it reports.
If it works has to be seen - whatever it is seems probably to have been rolled out here as well as on XF cloud, yet the guest numbers on here are still in the thousands:
Honestly, I hope they don't. Why aren't you expecting Linux distros to solve the issue? ISP's? There are far better people to solve such issues than end of the line software developers. This is the equivalent of asking every CMS, website stack, forum, etc, to solve the internet bad bot behaviour problem. Back to add-ons, this is what the developer market is for, for devs to build plugins for users to solve problems outside the scope of the base software. Third party DNS exists to meet this market.
For one simple reason that I have outlined ages ago already (and that you yourself have already proven just today but obviously failed to realize): Traditional approaches are typically IP based in one way or another. They are helpful, but - as we can see from the residential proxy problem - can only solve issues up to a certain point. From there you need behavioral analytics and for those to use the full set of possibilities those need knowledge about the application, its contents, its typical usage patterns and other things like that. While this analysis and deflection has not necessarily to happen at the border or within the application (and rather should not from a classical architectural perspective) the knowledge about patterns and behaviour has to come from the application. Who, I ask you, would be better suited to enable this than the people that built that application and for that reason know it inside out and can deal with it in a responsible way w/o collateral damage, better than any add on dev could?
Cheap answer. Add Ons are good for things that only a fraction of people need (and that therefor for all others would be unnecessary bloat). Things that everybody or almost everybody needs should be part of the core as every add on rises the complexity of the system, rises the risk for issues and conflicts and makes keeping the software up-to-date harder. As intelligente behavioral analysis is -for the time being - the relevant key for successful bot protection basically everyone needs it and it should for this reason be part of the core, at least in form of an API that others can build upon. Also, as this is a very sensitive area that can easily be broken through updates or influence performance negatively I think this should rather be implemented by the people who build the application than by 3rd party devs in form of an add on. The more, as it can be way better integrated during development then.
Sure. But as possibly everyone has recognized these classical ways of protection are no longer good enough - they miss a very relevant part of the problem and this is due to a structural problem they cannot solve.
Cheap argument and you claim I'd expect something I've never expected - I'v rather said quite the opposite and not just since yesterday. I've been talking about a multi-step setup for ages. It is not either/or, multistep is having both. Obviously one should filter as distant from the application or hot path as possible. But with traditional setups you will only manage to shrink the issue, not to solve it and you will miss quite a bit of bots/unwanted traffic. Still it is beneficial b/c this enables you to look closer at what's left at a second stage w/o overloading the application.
This is incorrect. You’re broadly right about the shape of the problem, but not about where the responsibility or solution sits.
But they are good enough. That's what we've proved just by adding a couple of rules to Cloudflare?
Well, it's solved here without further intervention needed. I'm not aiming to get the guest counts to zero, obviously. Yes, some bots might come through, but that doesn't matter. Cumulatively they're failing to have an effect and that's what matters. Spending engineering time on solving the problem at the application layer which might itself end up getting defeated is a waste of time and the fool's errand.
Privacy concerns too. Most of them wouldn't be GDPR / privacy law compatible as they set long cookies and do fingerprinting that is undesirable. Not really something we could even legally condone in code that we ship.
Definitely that is the proper way to deal with these rogue bot traffic issues - you want to at each level filter out the bad so the least amount of them hit your application.
+1 utilising the many tools prior to hitting the application like Cloudflare, logs etc helps. Though would l do like one addons ideal of allow bots to transverse a lighter PHP-using path than regularly visitors so bots don't queue up and choke PHP and thus MySQL server which is the weak point usually. Or even a path that doesn't involve PHP/MySQL and is cached/precached at webserver/redis/memcached level.
I do think some investigation into what we can do to improve guest page caching might be worthwhile. There is probably still some heavy work along that path which, if we can optimise and make it easier/clearer/document approaches for serving at the edge, or similar, might be worthwhile. Even if it was a manual thing where it's possible to enable it when stuff starts hitting the fan and it can be more aggressive in serving something very lightweight at the expense of perhaps being more stale.
No one has ever questioned that, so that is a bit of a straw man.
That wasn't Anthony's point at all. His point is performance and his argumentation, when you follow the thread, is "I dislike bots b/c they influence performance but I can deal with them by tuning my server performance. I don't care for the scraping b/c I consider the data public anyway and they may do with it whatever they like." Basically his motivation to lock out bots is perfomance, not protection of content.
In the meantime my list has grown to blocked 570 ASNs (including resident proxies) and somewhat around 75 blocked countries and I am still a happy camper.
Well, depends what you consider "good enough". A couple of thousand bots on my forum would clearly not be good enough for me. But as said: We have different goals.
I accept this answer - still I think you are missing out a big opportunity here. Not saying it would be trivial but if it worked it would be brilliant. However, given the challenges that XF faces regarding releases probably a bit of a dreamworld anyway.
I never claimed it was questioned.
This was the point I was referring to that was valid. The problem should be solved way before it hits any software. But providers are letting it run rampant and leaving it up to the guys at the end to deal with. There are serious abuses of internet protocols and Cloud services that are not being dealt with.
Yeah sorry for not reading nearly 1,000 posts, I don't have the time for that, unfortunately.
Some real human users aren't, I'm not sure you should be happy with the way you have things configured currently:
So even more the case that it's on you to implement the solutions you feel are necessary for your specific concerns. XenForo isn't going to help you protect your content. The web being crawlable and searchable and indexable is part of the normal state of affairs. Most customers want their content to reach as many people as possible, and privacy isn't really something that is relevant to this discussion. It might be for you, but that's just another example of the need for you to implement your own solutions and to stop expecting the software to do it for you.
No I'm not.
The behavioural data you're talking about, once again, and I wish I didn't have to labour the point, isn't needed to solve the problem. We've proved that here, and I'm proving it across XenForo Cloud sites too.
This isn't a measure of whether traffic should be permitted to load a thread in the slightest. Search crawlers will routinely hit threads that are ancient, most people wouldn't block that as this years or decades of content might be what drives the next top poster to their forum. Indeed, residential users might also find an old bookmark or a search result for a thread that is ancient that they need to see, but being met with a massive 403 Access Denied might put them off. There's no differentiator there, and the differentiators don't come from application specific stuff at all, they come from the behaviour of the request and the client in question, and whether they're hitting an old thread, new thread, a resource, a search or a random HTML page that doesn't exist, it's not something that is surfaced through XenForo, it's something that is surfaced through access logs and understanding how to find and categorise that information effectively. And at that point, that's where you can start challenging upstream at the edge or even just a simple rate limit. If you can't do that because of constraints of your hosting, that's not an application problem, it's a you problem.
As I said, most of the guests will likely be legitimate, the numbers aren't considerably higher than they ever were. Your goal matters to me (and most people) a lot less than you probably expect. You can push it all you like, but it's not the consensus concern across the sheer majority of customers, I promise you.
I'd challenge you on this that it seems rare for you to accept answers. I'm sure I'll get another encyclopaedic response to this because you don't seem to like people not agreeing with you. There's no big opportunity to be had here for us. The biggest opportunity is customers either learning to manage their servers effectively with the tools that have been working fantastically and improved for decades, or delegating the work to someone else who can.
This is something that I've pondered myself over the past couple months. XenForo is fast, exceptionally good at request comes in -> data goes out response times. Despite that, there are still physical limitations to what the CPU, RAM, Disk, NIC, etc. all can push in and out at any given moment. Under extreme load events, based on the type of a server handling the serving of content, things can get crippled.
style.php, ./js/*, among a few others. We can already use the built in AWS/Block store feature set for static user-based objects (avatars, attachments, etc.). However, greatly expanding such block storage support to the other portions of XenForo would be an excellent start at reducing overall server load. Sure, I could apply overrides myself on my forum, but having it natively built in would be ace.Absolutely. But it hasn't happened in the past and it won't happen in future. There is too much money to earn for those who would have the leverage and one better should not hope for legislation or politics when it comes to complex technical issues, even international ones. The times of the self regulating internet when you could simply call a phone number from a whois record and things were solved on the spot are long gone. These were good times.
Thanks for stopping by. ;-) Yes, I am happy, thanks for asking. You are now officially collateral damage. The audience of my forum does come to 99,5% from the DACH area plus a bit of Benelux. We do have a hand full of genuine guests from the UK, but very few. As some UK internet providers were pretty nasty at some point regarding residential proxies those have been blocked. Logged in users can bypass the restrictions and for some users I built dedicated exceptions. As I said: I block pretty brutally and I have the luxury to be able to do that. Due to the limitations on my shared hosting I overblock in some countries to keep things simple - on the new setup I may lower the bar for some of the UK providers and see how it goes.Some real human users aren't, I'm not sure you should be happy with the way you have things configured currently:
Just vanilla Chrome from a normal ISP btw. The bot problem is easy if you just 403 everything. Something that I cannot do for several hundred servers from all over the world.
Wow, that is a bold statement. Thanks for pointing this out clearly. I assume I am not the only one who sits in this boat.
We do no longer live in 2002, sadly. AI crawlers have changed the rules of the game and it is no longer about "crawlable and searchable" but about "stealing your content, sell it for profit and stop people from visiting your forum". Interestingly, @Anthony Parsons didn't get that point either. Look here, just a couple of posts back:
I think wider parts of the IT security industry that have been busy creating and selling things like intrusion detection tools and and other sensor based behavioral stuff would disagree with you here.
It is one example for a tiny data point among hundreds. Not the only one and not the one that would alone foster a decision. Regarding search crawlers: don't be silly. You know and I know that it is easy to verify them and let them pass and that is what basically all protective solutions do and have to do already.
I don't "expect" it - I just think it might be a good idea. Your opinion is obviously different and this is fine.
I guess if someone starts a startup with this business case he will have no issues collecting enough venture capital to have a nice life and maybe he would even create a product that becomes as successful as Cloudflare - just that such a person would not focus on XF as a target platform as it is too much of a too small niche. And I also guess that a couple of security companies work on things like that since ages - but now have the market to sell it and the technology to build it. In the enterprise market it is not even relevant if it really works and really brings benefit.
