Captain Obvious stepping in: source-based filtering (individual IPs, or broadened to entire subnets or AS numbers of ISPs, or countries) might be a good blanket filter for sources from where you do not expect any legitimate traffic, or the exceptions are rare enough to not be worth the trouble to give them a hall pass of sorts. But it is inadequate for residential proxies that mix with your normal visitors. Those need to be differentiated by behavior or other common characteristics.
The user-agent is useless, and will continue working for the diminishing minority of bots that still follow it. At best it becomes a way to whitelist the specific bot traffic you might still want to allow (for example DuckDuckGo spiders), but only in conjunction with also checking the source IP (so that you don't allow bad bots falsely claiming to be Google but coming from Tencent).
User logged in cookie might work for a while, until the bots discover that providing a fake cookie lets them through the network filter rule. Then they end up on the forum itself, where the next level of filtering would be applied. One variant is to ignore the fake cookie mismatching the user hash and serve the requested page as guest, the other is to throw an error about the invalid cookie. No idea how XenForo behaves at this moment, but either scenario is consuming some CPU and queries to the DB, so they'll have to be streamlined.
Another possible option is for the forum to allow a limited low number of daily page views per IP as guest, after which the visitor is redirected to the registration page. If implemented properly with server-side tracking (not cookie-based which can be tricked), it could partially avoid the load on the server instead of delivering hundreds or thousands of pages per day to some scraper bot leveraging the default unlimited-free-to-read permissions of the Guest/Unregistered group.
Considering the load on the server:
- The lowest burden on the server is to block specific traffic upstream.
- Somewhere in the middle are caching mechanisms for guests to receive pre-generated static pages that do not go through most of the PHP scripts and database queries, and are refreshed once every X minutes and only if the content has changed. A limited number of "free" unregistered page views per day per IP would limit this even further.
- The highest burden on the server is for one guest request to consume several network ports, to execute a bunch of PHP scripts (depending on CPU), to run several database queries (depending on CPU and disk speed), and to generate a page every single time a guest asks for it.
The typical bottleneck on servers is on the slow disk input/output for database queries. The difference between NVMe SSD, SATA SSD and spinning disks can be directly correlated with how responsive a site might seem, but it also depends on the hosting's capacity planning - there's a difference between 200 or 5000 clients and their sites competing for the same bandwidth of disk access, and the only option in your control might be to move host.
But with that bottleneck resolved, the second unavoidable bottleneck is the TCP/IP stack of communication ports. A server can basically use a maximum of roughly 64 thousand network ports on its IP to communicate with clients, which are kept open from the moment a client initiates a connection with a query until the server puts together the response (serving static files from disk or cache or assemblying a response through PHP scripts and database queries) and that response is transferred in its entirety back to the client, then the connection is closed and the port can be reused for the next request. Minimizing that time between request and finishing the response through various filters and optimizations will improve the number of overall requests the server can handle per minute, but at any given time one server cannot have more than about 64 thousand active connections.
Consider also that a typical visitor's browser will launch tens or >100 connections just for viewing just one page - one is the HTML of the page, the others are for CSS files, javascript files, any embedded images or anything else delivered from the server. So 64 thousand connections could realistically mean just a few thousand real visitors, before your server is overloaded. The options are then to trim down on the illegitimate requests and/or to add more servers behind a load balancer so that overall capacity is now in multiple of 64 thousand parallel connections. But that's going beyond the typical hosting category
[edit] Firefox's Inspector > Network shows that loading this page made nearly 40 connections to the XenForo server: 1 HTML, 11 CSS, 14 JS, 12 images. And a bunch others to gravatar, some CDN contents and a few onedollarstats.