Rebuilding a big forum takes DAYS

NealC

Member
I installed XF 2.1 as I'm new to XF and imported my big board from https://forums.flightinfo.com which was vB 4.2.x. Took about 24 hours for the import. I then installed the purchased enhanced search and did the rebuild. It took about 48 hours or more to complete the rebuild. Not sure if you can background the rebuild so we can launch and leave or do something to improve the rebuild time required.
 

Walter

Well-known member
Misconfiguration or slow server. One of my forums has twice your posts and Elasticsearch only took a few hours.
Rebuild in the background is fine, however your users won't be able to search.
 

NealC

Member
Misconfiguration.
I installed Elastic Search from the MSI installer in Windows. Other forums are fine. What may I have misconfigured if you would be so kind to assist in correcting the problem I may be seeing?
 

Kirby

Well-known member
There must be a serious misconfigutration or extremely underpowered hardware.

We did a an elasticsearch index rebuild (CLI! Web would take forever and a day, don't do that) for a XF 2.1 with 7.5M posts yesterday in 25 minutes.
 

NealC

Member
Probably something misconfigured, hardware is powerful, dual proc, 128 GB of RAM, raid 5, etc. Not a hardware issue. I changed the heap from 2048m to 4g and will see if that helps, not sure what else to change in the configuration as it's a stock install of ElasticSearch 6 on Windows Server 2012. Default setup from XF 2.1 as well.
 

Kirby

Well-known member
Don't do RAID 5, better go with RAID 10.
Yes, you will loose some capacity but disks are cheap!
 

NealC

Member
Don't do RAID 5, better go with RAID 10.
Yes, you will loose some capacity but disks are cheap!
I agree, just can't do it on this server. All new servers provisioned are RAID 10. My SAN is RAID 10, would love to offload all of this to the SAN and may someday.
 

andybond

Well-known member
I agree, just can't do it on this server. All new servers provisioned are RAID 10. My SAN is RAID 10, would love to offload all of this to the SAN and may someday.
I highly doubt you are saturating the fabric / san. I would wager there is something that needs optimising in the IIS layer.
There are things of course we can do other than RAID10 it , but even so , I doubt its needed. Block sizes , trespass ( ALUA etc ) but I feel you might be better of using perfmon and tracing where the bottleneck is.

What PHP version are you using?
 

NealC

Member
PHP 7.1. I think this is an ElasticSearch config issue as it's stock install until now changed memory to 4g. All other sites perform great, I don't think it's IIS, server, network, etc.
 

andybond

Well-known member
Just for testing - upgrade to PHP7.3.1 ( make sure its 7.3.1 , not 7.3 ) and elastic search 6.6.1?
 

NealC

Member
How can I measure ES performance after making a change? I certainly don't want to rebuild the search cache again. Is there any other way to tune and test?
 

Rudy

Well-known member
I've indexed ~15 million posts in maybe an hour and a half, on a rather modest dedicated server. I'm still not thrilled that ES has to use Java (which I hate with a passion--it is a resource hog compared to anything else I've ever used), but it works. I can't even remember what I configured for memory usage, but it was under what they recommended and it's been running fine for years.
 
Top