Fixed xf-rebuild:search --truncate fails with out-of-memory error since XenForo 2.3.0 Final

Steffen

Steffen

Well-known member
Affected version
2.3.0
After upgrading from XenForo 2.3.0 RC5 to XenForo 2.3.0 Final, xf-rebuild:search --truncate got killed for me (twice) because it used too much memory:

kernel: Out of memory: Killed process 459314 (php) total-vm:21573920kB, anon-rss:21323748kB, file-rss:640kB, shmem-rss:0kB, UID:1000 pgtables:42208kB oom_score_adj:200

XenForo 2.3.0 Final contains this change from 2.3.0 RC5 (probably to fix this bug):

Diff: 
--- a/src/XF/Job/AbstractJob.php
+++ b/src/XF/Job/AbstractJob.php
@@ -99,7 +99,7 @@ abstract class AbstractJob
 
         if ($percentSpent > 1)
         {
-            return max(1, $newExpected);
+            return max(1, min($maxBatch, $newExpected));
         }
 
         if ($remaining < 1 || $percentSpent >= .9)

After reverting this change, the rebuild worked as expected.

Maybe it needs to be checked that $maxBatch is not null or 0 (like elsewhere in this function)?
 
Just ran vanilla 2.3.0 code again with an added var_dump call in the if-clause. The issue is indeed that $maxBatch is null and min(null, $newExpected) is null. The function then returns 1 and it seems like this tiny batch turns into a problem.

Code: 
php cmd.php xf-rebuild:search --truncate
Rebuilding... Search index (Direct message 701)
Rebuilding... Search index (Direct message 12355)
NULL
Rebuilding... Search index (Direct message 116128)
Rebuilding... Search index (Direct message 116129)
(no more output follows until the out-of-memory error after several minutes)

Possible fix:
Diff: 
--- a/src/XF/Job/AbstractJob.php
+++ b/src/XF/Job/AbstractJob.php
@@ -99,7 +99,7 @@ abstract class AbstractJob
 
        if ($percentSpent > 1)
        {
-           return max(1, min($maxBatch, $newExpected));
+           return max(1, $maxBatch ? min($maxBatch, $newExpected) : $newExpected);
        }
 
        if ($remaining < 1 || $percentSpent >= .9)
 
Unfortunately, the proposed fix did not work on our server. Our forum has more than 20 million posts, and it is kind of useless without the search engine. The PHP-script now just eats a lot of RAM before it times out. I guess more than one of the important files has been changed since 2.2.16, when rebuilding the search index worked perfectly. We had to truncate the index after upgrading to XenForo Enhanced Search 2.3.0.
 
Last edited:
Having the same issue since upgrading, tried working with our host to see if we could solve it but there was nothing they could do. This is what they said:

It's not the wait timeout or anything like that because it was failing at ~1:30 or so.

I adjusted the max packet size to 1GB from 256MB for SQL and it went from failing at ~1:30 every time to failing at ~8:46. I suspect if I run it again it would fail at around the same time.

Just in case XenForo wants to know: Server version: 10.6.18-MariaDB.
Click to expand...

We have over 4 million posts, if that's useful info
 
I have found that the xf-rebuild:search process launches queries to the DB like this one, which consume a lot of RAM resources as it has a LIMIT of more than 100 million records!!

Code: 
SELECT xf_conversation_message.*, xf_conversation_master_Conversation_1.*
      FROM xf_conversation_message
      LEFT JOIN xf_conversation_master AS xf_conversation_master_Conversation_1 ON (xf_conversation_master_Conversation_1.conversation_id = xf_conversation_message.conversation_id)
      WHERE (xf_conversation_message.message_id > 451658)
      ORDER BY xf_conversation_message.message_id ASC
LIMIT 184086772;

While waiting for a new fixed release, Is there a way for xf-rebuild:search reading DB records in a paginated mode?
 
A quick fix would be to revert the file /cmd.php and the directory /src/XF/Job back to 2.2.X, from backup, and it will then work perfectly again. Please let me know if you need the files.
 
multibam said:
A quick fix would be to revert the file /cmd.php and the directory /src/XF/Job back to 2.2.X, from backup, and it will then work perfectly again. Please let me know if you need the files.
Click to expand...
Thank you very much for your help @multibam , but I'm a bit afraid of mixing code from different versions. Considering that there was a warning that there was no way back after upgrading I would only dare if someone from the development team vouched for it.... ( @Paul B ?)

But again, thank you for your response.
 
Thank you for reporting this issue, it has now been resolved. We are aiming to include any changes that have been made in a future XF release (2.3.1).

Change log:
Fix an issue where the batch size for search rebuilds could grow unbounded
Click to expand...
There may be a delay before changes are rolled out to the XenForo Community.
 
Back
Top Bottom