After upgrading to XF 2.2 and PHP 7.4, my server load increased, anyone else noticed it?

[sbj]

Graph for last 30 days.

I upgraded my forum from XF 2.1 to 2.2 and PHP 7.3 to 7.4.

Since then I see higher CPU load.

Before the average for my load was like between 0.05-0.20. Since then it is like 0.20-0.40.

There are also 3 high spikes but they may be explained I think. But you see that the average is much higher than before. It is not that big of a difference but still it is. So I wanted to ask if anyone else noticed such increase?

 

[sbj]

To be fair, I also upgraded a lot of addons to their 2.2 versions. Maybe couple of them added to the load? Not sure how to investigate this.

 

[Rhodium]

Check the queries in dev mode, maybe some addon is making this.

 

[Steffen]

We are seeing the same increase in CPU usage since updating to XenForo 2.2 on December 9th 2020:



Furthermore, the average response time as measured by Audisto has increased from 40 to 60 ms (overall, this average is considerably lowered by our own non-forum pages which are much faster than XenForo pages, I unfortunately don't have historical data of XenForo-only response times):



We only use official add-ons (XFES and XF301VB). We have heavily modified XenForo ourselves but nothing changed in this regard with the update from XenForo 2.1 to 2.2. Both XenForo 2.1 and 2.2 were/are running with PHP 7.4 and MySQL 5.7.

I tried to profile XenForo requests with XDebug but so far nothing has really come out of it. Maybe the templating system has become slower in XenForo 2.2 but that's a very wild guess at the moment.

 

[Steffen]

I think I've found the (or at least one) change that makes XenForo 2.2 much slower for us than XenForo 2.1: When I replace the body of the function "XF\Str\EmojiFormatter::formatEmojiToImage" with a simple "return $string" then I can speed-up one particular page I've tested locally (https://www.computerbase.de/forum/t...e-i7-und-i9-11000-im-desktop-benannt.2010111/) with Apache bench from 23 to 33 requests per second.

The mentioned function works with data provided by the JoyPixels library which got updated from version 4.0 to 6.0.1 in XenForo 2.2. In particular, its "unicodeRegexp" property which XenForo uses got updated. Replacing it with the value from XenForo 2.1 has the same effect as disabling the "formatEmojiToImage" function as mentioned above, it improves performance a lot.

The performance regression was introduced with JoyPixels 5.5: https://github.com/joypixels/emoji-toolkit/commit/af05016636de451c0bca8a2910fe503efff350e2

Why is a function "formatEmojiToImage" needed at all when my Emoji style is set to "Native device emoji"? Do you have any idea, @Chris D? It doesn't change one bit of the generated HTML in this case.

Patch (not yet tested much):

--- src/XF/Str/EmojiFormatter.php
+++ src/XF/Str/EmojiFormatter.php
@@ -26,6 +26,11 @@ class EmojiFormatter

     public function formatEmojiToImage($string)
     {
+        if ($this->config['style'] === 'native')
+        {
+            return $string;
+        }
+
         $client = $this->client;

         $string = preg_replace_callback('/' . $client->ignoredRegexp . '|' . $client->unicodeRegexp . '/u', function($match) use($client)

 

[StarArmy]

It might be a good idea to file this as a "bug report" in that forum so that XF's devs see this.

 

[sbj]

If Steffen wants to go ahead in reporting this, he can go ahead. I wouldn't know how to report this as a bug. I also have no data past 30 days. But good to know that I am not the only one experiencing this.

 

[Steffen]

I have implemented the change on our forums (https://www.computerbase.de/forum/) a few hours ago and will monitor the impact on the response times and server load in the coming days. So far it seems to have had a noticeable positive impact on server load but it's too soon to tell whether response times have been restored to XenForo 2.1 levels (the Audisto crawl I mentioned above takes multiple days and in turn impacts server load, so it's kind of a feedback loop).

Maybe this thread could just be moved to the bug reports forum? I don't know what else I could add at the moment, all the technical details I know are in post #5 above. If not then I'll report a bug once I have more data on server load and response times.

 

[sbj]

Thanks, will be looking forward to your feedback.

I would assist you, too if I only knew how to debug/investigate this as I don't know advanced server things to meassure things and even if I did, I wouldn't have any data for 2.1 anymore since I am already on 2.2.

Maybe based on your feeback we could ask this thread just to be moved like you suggest. We'll see.

 

[Steffen]

Average response time has dropped from 60 to 49 ms. That's not as good as with XenForo 2.1 (roughly 40 ms) but a big improvement.



Server load looks better, too:


I have filed a bug report here: https://xenforo.com/community/threa...brary-in-xenforo-2-2-with-partial-fix.192887/

 

[Kirby]

The mentioned function works with data provided by the JoyPixels library which got updated from version 4.0 to 6.0.1 in XenForo 2.2. In particular, its "unicodeRegexp" property which XenForo uses got updated. Replacing it with the value from XenForo 2.1 has the same effect as disabling the "formatEmojiToImage" function as mentioned above, it improves performance a lot.

The performance regression was introduced with JoyPixels 5.5: https://github.com/joypixels/emoji-toolkit/commit/af05016636de451c0bca8a2910fe503efff350e2

That Regex is insanely hard to read

But it might just be broken due to a little typo:

[\x{1F1E0}-\x{1F1FF}]{2}||[\x{1F468}-\x{1F469}\x{1F9D0}-\x{1F9DF}][\x{1F3FA}-\x{1F3FF}]?\x{200D}?[\x{2640}\x{2642}\x{2695}\x{2696}\x{2708}]?\x{FE0F}?

Note the double alternator ||, IMHO this doesn't make sense and effectively causes the regex to be way slower than the older version.
Regex101 does complain about this and doesn't even execute it.

Changing the double pipe to a single one seems to fix this.

 

[Mart]

This also fixed a 502 error I was getting following my recent update to 2.2.4 from 2.1.x. One of our threads has a lot of emoji in it (team flags) and one seemed to be causing this error. The thread was OK before the update and after applying the patch above (we're also using native emoji) it started working again.

My previous log searching just showed nginx throwing the error from php 7.2 fpm.

No definite conclusions here (I suspect other threads could also have been affected but this one is popular) but just in case anyone else is getting a mysterious 502 on certain pages I thought I'd mention it.

 

[dknife]

In case the bug thread isn't viewed, this is listed as fixed in 2.2.5

 

[rdn]

Note the double alternator ||,

Please guide us, on which file to modify on XenForo php files?
To temporarily patch this.

Thank you.

 

[Kirby]

The regex is in src/vendor/joypixels/emoji-toolkit/lib/php/src/Client.php line 25.

 

[Chris D]

For those not following the bug report or the release note announcements, this is fixed in XF 2.2.5 which we released yesterday.

 

[sbj]

Thanks. When I created this thread, I was not expecting it to be this relevant. Thanks to educated admins the issue seems to be found and it is now fixed by the XF crew. Thanks to you all!