Fixed CLI rebuild search OOM (MySQL backend)

[Xon]

Affected version

2.0.9

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 499712 bytes) in /var/www/sites/xf2.sufficientvelocity.com/html/src/XF/Db/AbstractAdapter.php on line 205

This appears to be occur when indexing ~10 million posts with the MySQL backend in debug mode, even with simple query logging (ie no stack trace) the amount of state which is tracked is enough to consume large amounts of memory (512mb memory limit).

In retrospect this isn't surprising, as the bulk-insert statement is being pushed through the query log system.

 

[Chris D]

Although this will predominantly affect the search rebuild, I think it's feasible that other rebuilds could cause issues, especially on servers with fewer resources.

So we've disabled query logging entirely by default but added a command option to switch it back on, e.g.

xf-rebuild:search --log-queries true

Worth noting, of course, it's possible to reduce the default batch size but at least now that shouldn’t be necessary unless there’s a specific need to.