XF 2.0 How long should a search index rebuild take?

[fotografik]

I know I know - how long is a piece of string.

After our migration from VB4 a couple of days ago, we ran into issues with ElasticSearch and Search in general. We have initiated a search index rebuild via CLI but after approximately 18 hours, we are only at Post #19782 out of 9.2M.

What could be the bottleneck and how can we speed the search index rebuild up?

 

[AndyB]

My site only has 1.6M posts. Elasticsearch index takes about 10 minutes to rebuild via browser not CLI.

 

[S Thomas]

What are your server specs?

 

[fotografik]

We are on an AWS RDS instance (db.m4.large) - 2vCPU and 8GB memory.

 

[S Thomas]

Are you using ElasticSearch, if so, how big is your heap size?

 

[fotografik]

Having some issues with ElasticSearch at the moment as it doesn't seem to be functioning. I get an error when I try to query its status - curl -XGET 'http://localhost:9200/_status'

 

[S Thomas]

What error?

 

[fotografik]

It’s working now. The curl command with _status has been deprecated which I only found out after some searching.
However, index rebuild is still taking a long time.

How can I check the heap size?

 

[S Thomas]

https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

 

[fotografik]

Appears that the heap size is set to 1GB.

Index rebuild is still very slow, and really at wit's end trying to understand where the bottleneck is. We were using SphinxSearch on VB4 previously and a reindex would take no more than 20 minutes at most.

 

[S Thomas]

Yep, 1GB is really not that much. It takes me ~15mins for 500k Posts, so that would be roughly 4,6h for your amount of posts with my server setup.
ES is very RAM hungry. Sphinx seems to be using standard SQL. So in that I guess you should disable Enhanced Search and try out indexing again. Or increase the heap size significantly.