XF 2.3 Problems rebuilding search index on a large site

Jon12345

Jon12345

Well-known member
Licensed customer
I just completed a migration from an old Knownhost server to a new Knowhost server. The elasticsearch was setup on the new server. But, when I try to rebuild the search index via cli or the Xenforo backend, it ends up crashing. It could be a RAM issue perhaps.

I've been using: /usr/local/php83/bin/php cmd.php xf-rebuild:search

Is there any way to do this without running out of RAM? Or is it some other issue?

I have 1.7M posts.
 
Sort by date Sort by votes
Jon12345 said:
Is there any way to do this without running out of RAM? Or is it some other issue?
Click to expand...
Probably would be a good idea to include how much ram you've allocated and perhaps the settings you're using for Elasticsearch. (total ram on server and what you have allocated in the ES server config, etc.) And what's the evidence of the crash? A log entry? or are you observing the ES instance crash out directly?

It very well could be a ram issue, it can get quite thirsty. And just to check all of the boxes, double check your available disk space.
You'll see in other information that the recommendations for ram are 50% of the server's memory with a max heap size of about 31gb.
 
Upvote 0 Downvote
Jon12345 said:
I just completed a migration from an old Knownhost server to a new Knowhost server. The elasticsearch was setup on the new server. But, when I try to rebuild the search index via cli or the Xenforo backend, it ends up crashing. It could be a RAM issue perhaps.

I've been using: /usr/local/php83/bin/php cmd.php xf-rebuild:search

Is there any way to do this without running out of RAM? Or is it some other issue?

I have 1.7M posts.
Click to expand...
Yeah this is pretty normal on a big site rebuilding the search index is one of the heaviest jobs XenForo does, so it can feel like it’s crawling or even stuck when it isn’t.

First thing I’d say is don’t run it through the browser. Use CLI instead

Code: 
php cmd.php xf-rebuild:search

That alone usually makes a big difference as it avoids timeouts and overhead from the web side.
If it’s still slow, try increasing the batch size a bit.

The default is quite low, so on decent hardware you can push more items through each run and speed it up.

Also depends what you’re using for search. If it’s just MySQL fulltext and you’ve got a large amount of posts, it’s always going to be slow. That’s where most larger forums switch to Enhanced Search (Elasticsearch) because it handles indexing way better at scale.

If you’re already on Elasticsearch and it’s struggling, I’d be looking at server resources or timeouts (memory, CPU, or even connection issues if it’s not running locally).

Other than that, it mostly comes down to size of the forum + server performance. On big installs it can just take a while to pass through everything.
 
Upvote 0 Downvote
ENF said:
Probably would be a good idea to include how much ram you've allocated and perhaps the settings you're using for Elasticsearch. (total ram on server and what you have allocated in the ES server config, etc.) And what's the evidence of the crash? A log entry? or are you observing the ES instance crash out directly?

It very well could be a ram issue, it can get quite thirsty. And just to check all of the boxes, double check your available disk space.
You'll see in other information that the recommendations for ram are 50% of the server's memory with a max heap size of about 31gb.
Click to expand...
My hosts support team says there isn't enough RAM. I currently have 6GB. Since I only really need to build it once, I presume I can just upgrade my hosting to 8b RAM, rebuild the cache, and then downgrade again to 6gb RAM. Or at least I hope that would work. Crash evidence is in terminal it says killed. And site freezes, until killed appears. I used AI to reduce the amount of RAM elasticsearch uses, but it was not enough.
 
Upvote 0 Downvote
Brad P said:
Yeah this is pretty normal on a big site rebuilding the search index is one of the heaviest jobs XenForo does, so it can feel like it’s crawling or even stuck when it isn’t.

First thing I’d say is don’t run it through the browser. Use CLI instead

Code: 
php cmd.php xf-rebuild:search

That alone usually makes a big difference as it avoids timeouts and overhead from the web side.
If it’s still slow, try increasing the batch size a bit.

The default is quite low, so on decent hardware you can push more items through each run and speed it up.

Also depends what you’re using for search. If it’s just MySQL fulltext and you’ve got a large amount of posts, it’s always going to be slow. That’s where most larger forums switch to Enhanced Search (Elasticsearch) because it handles indexing way better at scale.

If you’re already on Elasticsearch and it’s struggling, I’d be looking at server resources or timeouts (memory, CPU, or even connection issues if it’s not running locally).

Other than that, it mostly comes down to size of the forum + server performance. On big installs it can just take a while to pass through everything.
Click to expand...
Yes, done via CLI. Need to upgrade the RAM just to do the rebuild. Then lower it again.
 
Upvote 0 Downvote
what does your jvm.options look like? there's probably ways to not have to buy resources. you just need to spread the load out over more time and less concurrency.
 
Upvote 0 Downvote
Jon12345 said:
I've tried that with no luck. It seems to stop at the same post. Post 40495. Could it be a corrupted post causing the issue?
Click to expand...
Do you see a server error at admin.php?logs/server-errors/, when the rebuild job stops?

Do you see anything special, when you go to posts/40495/ in your browser?
 
Upvote 0 Downvote
briansol said:
it's very possible. what's in the post? special character? some weird image? there's probably something about that post.
Click to expand...
I deleted the post and still had the issue. My host says it is consuming 80 - 90% of server memory. The next step up in their hosting plans is 10GB RAM. Upgrade for one month, do the rebuild, then downgrade?


Code: 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                
1267327 accesswf  20   0 3211784   2.9g  30216 R  98.3  50.7   0:14.39 /usr/local/php83/bin/php -d memory_limit=1G cmd.php xf-rebuild:search --b+
1028212 elastic+  20   0 4808468 773228  18688 S   0.3  12.8  47:33.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -D+
1242903 mysql     20   0 2369680 168996  13964 S   2.6   2.8  37:11.15 /usr/local/mysql/bin/mysqld --datadir=/var/lib/mysql --socket=/run/mysql/+
1267312 accesswf  20   0  324612  75468  62320 S   3.0   1.3   0:03.87 php-fpm: pool accesswf                                                  
1266928 accesswf  20   0  324792  72008  58300 S   4.6   1.2   0:16.83 php-fpm: pool accesswf                                                  
1267287 accesswf  20   0  249808  71276  58740 S   1.3   1.2   0:04.65 php-fpm: pool accesswf                                                  
1266763 accesswf  20   0  325224  70176  57500 S   3.6   1.2   0:27.42 php-fpm: pool accesswf                                                  
1266910 accesswf  20   0  324996  69032  56440 S   2.6   1.1   0:16.20 php-fpm: pool accesswf                                                  
1266777 accesswf  20   0  324836  65440  54496 S   1.3   1.1   0:26.36 php-fpm: pool accesswf                                                  
1266772 accesswf  20   0  249708  65324  54032 S   5.0   1.1   0:26.52 php-fpm: pool accesswf                                                  
1266904 accesswf  20   0  249524  63260  52776 S   3.0   1.0   0:18.32 php-fpm: pool accesswf                                                  
1266696 accesswf  20   0  323856  63124  52776 S   2.0   1.0   0:32.63 php-fpm: pool accesswf


Code: 
[accesswf@server forums]$ /usr/local/php83/bin/php cmd.php xf-rebuild:search --batch 100
Rebuilding... Search index (Direct message 147)
Rebuilding... Search index (Direct message 25833)
Rebuilding... Search index (Direct message 55569)
Rebuilding... Search index
Rebuilding... Search index (Direct message reply 68620)
Rebuilding... Search index
Rebuilding... Search index
Rebuilding... Search index (Post 40498)
Killed
[accesswf@server forums]$


Code: 
[root@server ~]# free -mh
               total        used        free      shared  buff/cache   available
Mem:           5.8Gi       5.1Gi       172Mi        46Mi       740Mi       622Mi
Swap:          4.0Gi       3.1Gi       894Mi
 
Upvote 0 Downvote
briansol said:
what does your jvm.options look like? there's probably ways to not have to buy resources. you just need to spread the load out over more time and less concurrency.
Click to expand...
As @briansol says, what is your heap size in the options file?

Looks something like:
Code: 
-Xms1g
-Xmx1g

Lower the ram to something like:
Code: 
-Xms512m
-Xmx512m

Reload Elasticsearch and try again
 
Upvote 0 Downvote
I upgraded from 6gb to 10gb for one month. I ran the CLI rebuild and it took about 5 minutes. I hope that is a good sign, not a bad sign!

Code: 
root@server forums]# /usr/local/php83/bin/php cmd.php xf-rebuild:search
Rebuilding... Search index (Direct message 809)
Rebuilding... Search index (Direct message 40133)
Rebuilding... Search index (Direct message 55571)
Rebuilding... Search index
Rebuilding... Search index (Direct message reply 68624)
Rebuilding... Search index
Rebuilding... Search index
Rebuilding... Search index (Post 40498)
Rebuilding... Search index (Post 1988935)
Rebuilding... Search index
Rebuilding... Search index (Profile post 5)
Rebuilding... Search index (Profile post 3388)
Rebuilding... Search index (Profile post 4436)
Rebuilding... Search index
Rebuilding... Search index (Profile post comment 253)
Rebuilding... Search index
Rebuilding... Search index (Thread 114858)
Rebuilding... Search index (Thread 114859)
Rebuilding... Search index (Thread 335390)
Rebuilding... Search index
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

iaresee
  • Question Question
XF 1.5 Converting YouTube, SoundCloud links on a large site
Replies
1
Views
944
iaresee
iaresee
Kier
Rebuilding the Search Index for a Large Site - Experts Only
Replies
12
Views
9K
Douglas Taylor
Douglas Taylor
Back
Top Bottom