Fixed Upgrade taking many hours (phrase import)

ZuluView

Member
So far it's been running for 4 hours and I'm at "Importing... Master data (Phrases: 4.4%)" and looks to be taking around an hour for each %.

Is this something others have experienced?

Did the upgrade via the button in the admin panel.
 

Mike

XenForo developer
Staff member
If I’m correct, your server just updated its MariaDB version didn’t it? I think there might be a regression in this version with a particular query that is used by the phrase system here (which has a very large list of phrase names in an IN clause).

I’m not sure if it’s possible to temporarily roll back the MariaDB change, to help confirm this. I did workaround this for another customer a recently, though at the time it appeared to be server-specific. We will likely need to change our approach for now, but if this is a performance regression in MariaDB, we will likely need to gather a reduced test case to report to them.

I’m going to move this to bugs for now. If you’re stuck with this and can’t/aren’t comfortable with server changes, let me know and I’ll see if I can pull a quick and dirty fix.
 

Mike

XenForo developer
Staff member
Note that this may manifest as a “server has gone away” error as well. It isn’t totally consistent.
 

Mike

XenForo developer
Staff member
Quick and dirty fix should be as follows.

In src/XF/AddOn/DataType/Phrase.php, find:
Code:
$existing = $this->finder()->where('title', $ids)->where('language_id', 0)->keyedBy('title')->fetch();

Replace with:
Code:
//$existing = $this->finder()->where('title', $ids)->where('language_id', 0)->keyedBy('title')->fetch();



In the same file, then find: (13 lines below)
Code:
$entity = $existing[$id] ?? $this->create();

Replace with:
Code:
$existingPhrase = $this->em->findOne('XF:phrase', [
   'title' => $entry['title'],
   'language_id' => 0
]);

$entity = $existingPhrase ?? $this->create();
 

XF Bug Bot

XenForo bug fixer bot
Staff member
Thank you for reporting this issue, it has now been resolved. We are aiming to include any changes that have been made in a future XF release (2.2.2).

Change log:
Break the phrase import query into chunks to avoid a MariaDB performance regression.
There may be a delay before changes are rolled out to the XenForo Community.
 

antonino

Member
Ιn my case and in the second part of your solution i see this in line 69...

Code:
$entity = isset($existing[$id]) ? $existing[$id] : $this->create();

Do i have to make the same changes you suggest?
 

Jared26

Member
Quick and dirty fix should be as follows.

In src/XF/AddOn/DataType/Phrase.php, find:
Code:
$existing = $this->finder()->where('title', $ids)->where('language_id', 0)->keyedBy('title')->fetch();

Replace with:
Code:
//$existing = $this->finder()->where('title', $ids)->where('language_id', 0)->keyedBy('title')->fetch();



In the same file, then find: (13 lines below)
Code:
$entity = $existing[$id] ?? $this->create();

Replace with:
Code:
$existingPhrase = $this->em->findOne('XF:phrase', [
   'title' => $entry['title'],
   'language_id' => 0
]);

$entity = $existingPhrase ?? $this->create();

Just wanted to thank you and say that helped me out of a tough spot.
 

Mike

XenForo developer
Staff member
Ιn my case and in the second part of your solution i see this in line 69...

Code:
$entity = isset($existing[$id]) ? $existing[$id] : $this->create();

Do i have to make the same changes you suggest?
Replacing that line with the one I mentioned should be fine (provided you're running PHP7, which you likely would be).

The changes I listed are based on the 2.2 version. The line you're showing would be from 2.1.
 

Mike

XenForo developer
Staff member

Wildcat Media

Well-known member
10.3.26 killed my XenForo and WordPress sites a few days ago when I ran my upgrades--I was able to roll back to 10.3.25 to get them back online. I saw their dismissive note abut it in the emergency release:

But regardless of where the bug is, from the user point of view it’s MariaDB upgrade that broke their applications. And they cannot always move to PHP 7.3 or wait for Oracle to fix connectors.

It was broken in PHP 7.4 also--I was already running PHP 7.3 so that shouldn't have been affected, but it was, and I tried upgrading to 7.4 nonetheless. No change. Just dropping this here in case anyone encounters this in the near future.
 

djbaxter

Well-known member
cPanel is automatically rolling out MariaDB 10.3.27 which fixes the problem if your settings allow for auto updates. Mine was updated Tuesday night (3 days ago now).
 
Top