XF 2.3 Performance improvements in XenForo 2.3.14

XenForo 2.3.14 is being released, all being well, next week.

In addition to the usual bug fixes and stability improvements, this upgrade also has a focus on performance. For quite some time we have all been dealing with what feels like a never ending influx of residential proxy bot swarms, DDoS attacks, aggressive crawlers (both AI and traditional) and scrapers.

Each time a request hits XenForo, the cost of this is a PHP worker connection, a database connection, CPU and memory, and unless your server is over-specced (they usually aren't) you have a finite pool of resources that real members are fighting for.

One of the biggest problems at the moment is automated traffic using residential proxy networks. This gives bad actors an upper-hand as the traffic often appears to be genuine, and human. You could ban an entire CIDR/AS Number that is being used for malicious purposes today, only to find that tomorrow those questionable IP addresses now belong to real users who cannot access your site.

Traditional detections and mitigations based on IP address alone are no longer effective. Attacks can send an extreme amount of volume and cause disruption with tens of thousands of IP addresses doing only one or two requests each.

In our recent experience, even Cloudflare and their premium mitigations for this, Cloudflare Superbot Fight Mode, are not always able to distinguish genuine human traffic from automation, and on their own, at least with default configurations, they are not enough. They are useful tools; if you're able to (or someone can on your behalf) dedicate hours per day towards monitoring your logs, detecting new patterns, writing rules to present challenges, while also trying to minimise the disruption to real humans, you're in a lucky position. But things move fast and it's difficult to stay on top for long, so we, as-in XenForo, now has to start looking inward at what we can do to lessen the impact and cost of some of these requests.

And that's what XenForo 2.3.14 is starting to do.

whats-new performance improvements​

Each guest visiting any whats-new page, including /posts, /profile-posts, /media, /resources now gets a single, shared, cached result that is returned from the root URL rather than returning a page generated from a specific numbered record. Every guest gets the same set of results and, if enabled, the HTML for those results is returned directly from the page cache. Care is also taken to ensure that under heavy concurrent traffic, there isn't a "cache stampede" effect when rebuilding the cached results. The cache records are now saved with the time the result was saved rather than from when the request started so that cached results are valid for longer when a site is under load and the stale cache records start taking longer to generate. "New profile posts" is also now constrained by the read marking data lifetime so less work is done to display this set of results.

Image proxy and oEmbed improvements​

Concurrent requests for the same image proxy or oEmbed record can no longer happen. If available, users are served the stale cache result until the new cached result is available, rather than multiple requests happening for the same resource concurrently.

oEmbed requests are now signed as previously oembed.php could fetch and store records for media that was not being shown on the page.

And the responses of image proxy and oembed requests can now be cached by browsers and CDNs more easily.

POST submission links now hardened​

Some URLs within XenForo, particularly the style, style variation, language and cookie consent URLs were previously GET requests with a unique CSRF token embedded within. On XenForo Cloud we see crawlers and bots following misc/style-varation links at a rate of over 10 million requests per-week, and similar volumes for the other URLs. Originally destined for XenForo 2.4, there is a new click handler you can opt into named link-post which submits a standard looking link as a POST request.

Limits on concurrent guest searches​

The number of searches that can be triggered by logged out users collectively is now constrained to a maximum of 4 concurrent requests at a time. Guests were triggering searches at a volume of 40 million requests per-week on XenForo Cloud. At that volume, searches were often causing PHP workers to become saturated and impact normal users.

Starting with XenForo 2.3.14, new installs will have guest searching disabled (via permissions) by default. Existing installations are not affected. However, we do still generally recommend disabling guest searching if it is not a relevant feature for your community.

"About" tab optimisations​

The way that the following and followers lists are generated for a member's "About" tab has changed to avoid using MySQL's RAND() function. This tab was expensive to run in some cases and was often being abused as such.

Better indexing for user-based searches for MySQL​

Some search requests that do not rely on keywords ("Recent content" and "Find all content by user") are now optimised to benefit from existing table indexes with PHP taking over some of the work to avoid full table scans for these specific searches.

Attachment caching​

Full-sized attachments are now cached for up to one-hour, previously not being cached at all. We were also touching the file-system or remote object storage for every attachment request to check for existence, even if it was going to be served from the cache anyway. Particularly for remote storage, this results in fewer existence checks which added time to those requests.

Guest page cache can now cache permanent redirects​

Previously, the guest page cache was only able to store and serve HTML and cached resulting from a successful HTTP code 200 request. In 2.3.14 we have now extended this to permanent redirects. This will apply to posts/<n> style links and goto/post/<n> style links which are computationally expensive as they have to find the thread, forum, specific page and generate the full URL before redirecting. These requests now store the original URL and its intended target so that the redirect can take place immediately without doing the additional work. On XenForo Cloud, hits to the common redirect routes were averaging over 15 million per week.

All of these changes are live now on XenForo community. Please open a bug report if you find anything that doesn't seem right when browsing as a guest.

If you are struggling with bots we highly recommend upgrading to XenForo 2.3.14 upon its release and continue to provide feedback on the types of traffic pressure you are seeing across the software. These changes are a good start but we do not view this game of whack-a-mole as being over by a long chalk. There'll always be more to do!

xenforo_performance_imrpovements_2_3_14.webp
 
Thank you. What's-New is one of the biggest targets. I found that 30% of the residential proxies going to that pathway can solve Cloudflare challenges. And some real users were failing Cloudflare challenges. So this should help a lot.
 
About time.
I hear it. It's taken a lot of work (and data) to understand the extent of the problem, while also being difficult to find the time to reach for the permanent solutions amidst the almost daily effort required to mitigate individual shapes of traffic causing problems in ways that don't get in the way of real users.
 
We've had good luck using Gemini (paid version) to analyze our http error logs, website traffic Logs, and PHP error logs. Many rewrite rules that abort probes and scraping have really helped! I also used it to tweak the My.ini and Php.ini file!

Websites load faster than ever on the server including XF.
 
I hear it. It's taken a lot of work (and data) to understand the extent of the problem, while also being difficult to find the time to reach for the permanent solutions amidst the almost daily effort required to mitigate individual shapes of traffic causing problems in ways that don't get in the way of real users.
Many of us have been plugging the holes by adding Cloudflare security rules left and right. A quick look at one of the most popular discussions the past few months can tell the story.
It's great to get these issues addressed at the core level but as you already realize, it's a long battle with these AI bots, spams. It's the time we live in.
 
Thanks for the update, Chris. It's a forever war with those malicious actors, isn't it? I'm glad to say that performance remains good on my Cloud service.
 
Starting with XenForo 2.3.14, new installs will have guest searching disabled (via permissions) by default. Existing installations are not affected. However, we do still generally recommend disabling guest searching if it is not a relevant feature for your community.
I disabled it on our various forum installs and nobody has complained.

I've installed an add-on that lets visitors search using Google. That at least gives them a search option that will keep them on the forum. Not ideal but it does keep the unwanted search hits away.

Does the What's New feature use the forum's search engine?
 

POST submission links now hardened​

Some URLs within XenForo, particularly the style, style variation, language and cookie consent URLs were previously GET requests with a unique CSRF token embedded within. On XenForo Cloud we see crawlers and bots following misc/style-varation links at a rate of over 10 million requests per-week, and similar volumes for the other URLs. Originally destined for XenForo 2.4, there is a new click handler you can opt into named link-post which submits a standard looking link as a POST request.
I implemented a variant of this for older XF versions and it does make a difference.

I'll need to adjust SV/StandardLib to avoid duplicating the XF2.3.14 feature but it is welcome to see that coming forward for XF2.3 :)
 
I've installed an add-on that lets visitors search using Google. That at least gives them a search option that will keep them on the forum. Not ideal but it does keep the unwanted search hits away.
I use the XF Enhanced search. Not seen any performance issues discussed here.

 
I use the XF Enhanced search.
Ours do as well. The earliest go back to 2011-2012. 😁 We all have different traffic patterns (and bot activity) so it was a preemptive move in a couple of instances.

We've also had member profiles inaccessible to the public for years. If they want to read them, they can join up.
 
@Chris D one thing I've found to help is to restrict thread filtering options for guests.

For example I've locked down the sorting option because it causes very different performance patterns and some of the sorting options are just too expensive these days (aka sorting by title).
 
Thank you for this effort.

We have been getting hit hard for months. Cloudflare was screening 575k of them yesterday while the forum itself still was dealing with 41k. It has been a whack-a-mole effort to keep sql up lately.
 
Back
Top Bottom