XF 2.3 Problems rebuilding search index on a large site

[Jon12345]

I just completed a migration from an old Knownhost server to a new Knowhost server. The elasticsearch was setup on the new server. But, when I try to rebuild the search index via cli or the Xenforo backend, it ends up crashing. It could be a RAM issue perhaps.

I've been using: /usr/local/php83/bin/php cmd.php xf-rebuild:search

Is there any way to do this without running out of RAM? Or is it some other issue?

I have 1.7M posts.

 

[ENF]

Is there any way to do this without running out of RAM? Or is it some other issue?

Probably would be a good idea to include how much ram you've allocated and perhaps the settings you're using for Elasticsearch. (total ram on server and what you have allocated in the ES server config, etc.) And what's the evidence of the crash? A log entry? or are you observing the ES instance crash out directly?

It very well could be a ram issue, it can get quite thirsty. And just to check all of the boxes, double check your available disk space.
You'll see in other information that the recommendations for ram are 50% of the server's memory with a max heap size of about 31gb.

 

[Brad P]

I just completed a migration from an old Knownhost server to a new Knowhost server. The elasticsearch was setup on the new server. But, when I try to rebuild the search index via cli or the Xenforo backend, it ends up crashing. It could be a RAM issue perhaps.

I've been using: /usr/local/php83/bin/php cmd.php xf-rebuild:search

Is there any way to do this without running out of RAM? Or is it some other issue?

I have 1.7M posts.

Yeah this is pretty normal on a big site rebuilding the search index is one of the heaviest jobs XenForo does, so it can feel like it’s crawling or even stuck when it isn’t.

First thing I’d say is don’t run it through the browser. Use CLI instead

php cmd.php xf-rebuild:search

That alone usually makes a big difference as it avoids timeouts and overhead from the web side.
If it’s still slow, try increasing the batch size a bit.

The default is quite low, so on decent hardware you can push more items through each run and speed it up.

Also depends what you’re using for search. If it’s just MySQL fulltext and you’ve got a large amount of posts, it’s always going to be slow. That’s where most larger forums switch to Enhanced Search (Elasticsearch) because it handles indexing way better at scale.

If you’re already on Elasticsearch and it’s struggling, I’d be looking at server resources or timeouts (memory, CPU, or even connection issues if it’s not running locally).

Other than that, it mostly comes down to size of the forum + server performance. On big installs it can just take a while to pass through everything.