Rebuilding a big forum takes DAYS

[NealC]

I installed XF 2.1 as I'm new to XF and imported my big board from https://forums.flightinfo.com which was vB 4.2.x. Took about 24 hours for the import. I then installed the purchased enhanced search and did the rebuild. It took about 48 hours or more to complete the rebuild. Not sure if you can background the rebuild so we can launch and leave or do something to improve the rebuild time required.

 

[Walter]

Misconfiguration or slow server. One of my forums has twice your posts and Elasticsearch only took a few hours.
Rebuild in the background is fine, however your users won't be able to search.

 

[NealC]

Misconfiguration.

I installed Elastic Search from the MSI installer in Windows. Other forums are fine. What may I have misconfigured if you would be so kind to assist in correcting the problem I may be seeing?

 

[Kirby]

There must be a serious misconfigutration or extremely underpowered hardware.

We did a an elasticsearch index rebuild (CLI! Web would take forever and a day, don't do that) for a XF 2.1 with 7.5M posts yesterday in 25 minutes.

 

[NealC]

Probably something misconfigured, hardware is powerful, dual proc, 128 GB of RAM, raid 5, etc. Not a hardware issue. I changed the heap from 2048m to 4g and will see if that helps, not sure what else to change in the configuration as it's a stock install of ElasticSearch 6 on Windows Server 2012. Default setup from XF 2.1 as well.

 

[Brent W]

Is there a specific reason you are using Windows to host your forums?

 

[Kirby]

Don't do RAID 5, better go with RAID 10.
Yes, you will loose some capacity but disks are cheap!

 

[NealC]

Is there a specific reason you are using Windows to host your forums?

All of my servers are Windows Server 2012, that's what my business uses. I'm a .NET developer.

 

[NealC]

Don't do RAID 5, better go with RAID 10.
Yes, you will loose some capacity but disks are cheap!

I agree, just can't do it on this server. All new servers provisioned are RAID 10. My SAN is RAID 10, would love to offload all of this to the SAN and may someday.

 

[andybond]

I agree, just can't do it on this server. All new servers provisioned are RAID 10. My SAN is RAID 10, would love to offload all of this to the SAN and may someday.

I highly doubt you are saturating the fabric / san. I would wager there is something that needs optimising in the IIS layer.
There are things of course we can do other than RAID10 it , but even so , I doubt its needed. Block sizes , trespass ( ALUA etc ) but I feel you might be better of using perfmon and tracing where the bottleneck is.

What PHP version are you using?

 

[NealC]

PHP 7.1. I think this is an ElasticSearch config issue as it's stock install until now changed memory to 4g. All other sites perform great, I don't think it's IIS, server, network, etc.

 

[andybond]

Just for testing - upgrade to PHP7.3.1 ( make sure its 7.3.1 , not 7.3 ) and elastic search 6.6.1?

 

[NealC]

How can I measure ES performance after making a change? I certainly don't want to rebuild the search cache again. Is there any other way to tune and test?

 

[Wildcat Media]

I've indexed ~15 million posts in maybe an hour and a half, on a rather modest dedicated server. I'm still not thrilled that ES has to use Java (which I hate with a passion--it is a resource hog compared to anything else I've ever used), but it works. I can't even remember what I configured for memory usage, but it was under what they recommended and it's been running fine for years.