Possibly a key point to remember is that we're not looking for A solution, a single, magical rule on the receiving end that would identify (and block) bots.
We have to think in layers. Have 5 or 8 or 20 different triage levels for all sorts of criteria, so that every time some of the traffic gets routed out. Lots of other activities follow the same approach: theft deterrent, or water purification, or patient intake.
Then the second step is to put those layers at their right places in the chain so that they are most efficient. You can do with software any kind of fancy stuff, but to do IP level filtering at the endpoint is the most "expensive" way of doing it instead of putting that filtering upstream into the transport layer where dedicated hardware can do it much, much better. I mean, you don't want all sick people entering the hospital door to go all the way to the sterile operating room and get their first check there; there's layers of checks, filters and decisions along the way, and for good reason.
So I tend to agree with the line of thought that XenForo should not step (too much) out of its intended purpose as a web app, and go nuts into becoming a security/privacy tool as well. I would expect that the forum admin buying the solution is given some guidance or best practices or at least some hints into setting up their server so that the forum experience is reasonably good, and that XF does have some built-in features to cater to the different options for every decision a forum admin needs to make. Like, if I want to allow or disallow some types of bots, how do I do that? If I want full, limited or no access for guests, how do I do that? If privacy/GDPR/anonymity is a concern, how do I tackle that? (I still have a GDPR issue with birthdays being stored in the database, for example.) Can I build my wall for the garden as a forum plugin if I want to, or can I use another component lower in the stack on my server or upstream in the communication path, or a combination of them? Do I have any flexibility to decide for myself and combine stuff, or do I absolutely have to fit within the rigid assumptions of the product?
However, although technically XenForo can be developed to incorporate stuff, I don't think it is the right approach for it to turn into a monolithic bundle of forum AND security AND who knows what else we want to throw at it then. Use a combination of different tools that are best for their limited purpose, deployed at the right place in the chain, and together they integrate to fit the need. Layers.
We could go down an endless philosophical debate about responsibility and where it fails in this chain from the forum as the end target and going upstream towards the source of the problem:
- Should the web application embed security features, or should there be two distinct components in the stack, one for the web app and another for the web app firewall?
- Should web hosts be responsible for hardening their servers, or why should it be tolerated that people with zero server admin experience or training can just rent VPS and introduce a vulnerable system in the internet?
- Should datacenters on the hosting side be responsible for ingress filtering and/or a way for customers to control the firewall rules there?
- Should transit networks have any responsibility in any security monitoring or at least an efficient, automated way for bad traffic to be reported and blocked?
- Should datacenters hosting bot farms be responsible to monitor their clients' software and use of infrastructure, proactively enforce their ToS and/or laws concerning abuse, and doing client screening before signing them up, and having effective methods to process the feedback that some clients are doing nasty stuff and terminate them (with penalties)?
- Should the people who buy "smart" devices have to take some training and pass some certification for their explicit responsibility coming with owning an internet-connected device, before they can buy said device?
- Should manufacturers of "smart" devices have to pass through some security certifications for their firmware and some legal obligations for long term security patching as part of their conformity approval for bringing a new product on the market?
- Should legislative bodies, governmental agencies and industry regulators do better at establishing and enforcing security and fair use of the internet?
Having just written that, I'm pissed to realize how little control do we actually have, how surprisingly long did the internet function relatively ok on such a rotten foundation, and how easy it is for unscrupulous agents to just do as they please with little to zero consequences.
One more thing, steming from decades of being a little cog in the machinery of Fortune 500 enterprises: whenever you face a crisis or incident, it is important to get onto fixing it, but it is MORE important to communicate about it. For a technically minded person it is f'ing annoying to have to interrupt the work on a fix to join war room calls or send e-mails or notifications about the issue, the progress, the projected road ahead etc. - that's why some companies' major incident management process pair up the technical skilled people with incident managers who only do the communication side of it. There is immense benefit from lots of other people being informed, not just from the problem being fixed.
So kudos to
@Chris D for the insane amount of work to stabilize the inflow of traffic abusing the hosted forums infrastructure. We share the pain (possibly not at that level, but this thread is evidence of how many people face the same issues) but I wish we joined forces earlier through open communication and figuring things together. Leverage the power of community facing a common problem, let them pitch in ideas and test things while you get some much needed sleep, then come back in the morning to a maybe half baked thing which might actually work for a while.
Neither of us is alone in this.
this guy really admins 
I might do both
