rebuild search index very slow

[KSA]

I have recently purchased enhanced search, installed elsticsearch, emptied search index, delete search index before rebuilding and still very very slow. Its very frustrating.

Any suggestion?

 

[Ozzy47]

Are you running MySQL for your database?

 

[KSA]

Yes.

 

[Ozzy47]

Perhaps this post from Mike will help.

 

[Mike]

You actually configured XFES and enabled it in the control panel right?

(The exact steps differ between 1.x and 2.x and I'm not sure which version you're on.)

 

[KSA]

You actually configured XFES and enabled it in the control panel right?

(The exact steps differ between 1.x and 2.x and I'm not sure which version you're on.)

I am on xf2 latest release/elastichsearch 6.4.2 and I did configure XFES and pretty much used the default values aside from checking empty search index and delete search index before rebuilding. Its been around six hours and only indexed 740000 out of 2.5M posts.

 

[Mike]

You should either change the number of items processed per page in the rebuild -- Elasticsearch can generally support much higher numbers than MySQL's full text search. (Think 5000+ per page).

Or you may wish to consider using the CLI rebuild via php cmd.php xf-rebuild:search which will avoid the web overhead.

If both of those options still seem very slow, then is Elasticsearch installed locally or on another server? If it's on another server, then it sounds like perhaps there's a network issue. If it's local, then you may need to look at whether Elasticsearch needs some amount of optimization.

 

[Xon]

Elasticsearch can generally support much higher numbers than MySQL's full text search. (Think 5000+ per page).

ElasticSearch actually likes ingesting large amounts of data at once vs individual rows, you get some hilarious disk-usage sprawl pumping small batches in due to the nature of the backend. Eventually it compacts down, but there is a performance hit for a while.

 

[KSA]

You should either change the number of items processed per page in the rebuild -- Elasticsearch can generally support much higher numbers than MySQL's full text search. (Think 5000+ per page).

Or you may wish to consider using the CLI rebuild via php cmd.php xf-rebuild:search which will avoid the web overhead.

If both of those options still seem very slow, then is Elasticsearch installed locally or on another server? If it's on another server, then it sounds like perhaps there's a network issue. If it's local, then you may need to look at whether Elasticsearch needs some amount of optimization.

It was 2500 per page and yes its on localhost. I have not tried to push it through the CLI. No optimization required. Just stopped suddlny while ago given this error

XFES\Elasticsearch\ConnectException: Elasticsearch indexing error: cURL error 28: Operation timed out after 20001 milliseconds with 0 bytes received src/addons/XFES/Elasticsearch/Api.php:368
Generated by:  Oct 8, 2018 at 7:19 AM
Stack trace
#0 src/addons/XFES/Elasticsearch/Api.php(403): XFES\Elasticsearch\Api->request('post', '_bulk', '{"index":{"_ind...', Array, Object(GuzzleHttp\Message\Request))
#1 src/addons/XFES/Elasticsearch/Api.php(171): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#2 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#3 src/addons/XFES/Search/Source/Elasticsearch.php(57): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#4 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#5 src/XF/Search/Search.php(59): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post))
#6 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('post', Object(XF\Mvc\Entity\ArrayCollection))
#7 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('post', 1495257, '2500')
#8 src/XF/Job/Manager.php(241): XF\Job\SearchRebuild->run(7.9999990463257)
#9 src/XF/Job/Manager.php(187): XF\Job\Manager->runJobInternal(Array, 7.9999990463257)
#10 src/XF/Job/Manager.php(103): XF\Job\Manager->runJobEntry(Array, 7.9999990463257)
#11 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#12 src/XF/Mvc/Dispatcher.php(249): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#13 src/XF/Mvc/Dispatcher.php(88): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', 'html', Object(XF\Mvc\ParameterBag), 'tools', Object(XF\Admin\Controller\Tools), NULL)
#14 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#15 src/XF/App.php(1931): XF\Mvc\Dispatcher->run()
#16 src/XF.php(328): XF\App->run()
#17 admin.php(13): XF::runApp('XF\\Admin\\App')
#18 {main}

-------------

Previous GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 20001 milliseconds with 0 bytes received - src/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:49
#0 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(103): GuzzleHttp\Exception\RequestException::wrapException(Object(GuzzleHttp\Message\Request), Object(GuzzleHttp\Ring\Exception\ConnectException))
#1 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(132): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#2 src/vendor/react/promise/src/FulfilledPromise.php(25): GuzzleHttp\RequestFsm->GuzzleHttp\{closure}(Array)
#3 src/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
#4 src/vendor/guzzlehttp/guzzle/src/Message/FutureResponse.php(43): GuzzleHttp\Ring\Future\CompletedFutureValue->then(Object(Closure), NULL, NULL)
#5 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(134): GuzzleHttp\Message\FutureResponse::proxy(Object(GuzzleHttp\Ring\Future\CompletedFutureArray), Object(Closure))
#6 src/vendor/guzzlehttp/guzzle/src/Client.php(165): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#7 src/addons/XFES/Elasticsearch/Api.php(351): GuzzleHttp\Client->send(Object(GuzzleHttp\Message\Request))
#8 src/addons/XFES/Elasticsearch/Api.php(403): XFES\Elasticsearch\Api->request('post', '_bulk', '{"index":{"_ind...', Array, Object(GuzzleHttp\Message\Request))
#9 src/addons/XFES/Elasticsearch/Api.php(171): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#10 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#11 src/addons/XFES/Search/Source/Elasticsearch.php(57): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#12 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#13 src/XF/Search/Search.php(59): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post))
#14 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('post', Object(XF\Mvc\Entity\ArrayCollection))
#15 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('post', 1495257, '2500')
#16 src/XF/Job/Manager.php(241): XF\Job\SearchRebuild->run(7.9999990463257)
#17 src/XF/Job/Manager.php(187): XF\Job\Manager->runJobInternal(Array, 7.9999990463257)
#18 src/XF/Job/Manager.php(103): XF\Job\Manager->runJobEntry(Array, 7.9999990463257)
#19 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#20 src/XF/Mvc/Dispatcher.php(249): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#21 src/XF/Mvc/Dispatcher.php(88): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', 'html', Object(XF\Mvc\ParameterBag), 'tools', Object(XF\Admin\Controller\Tools), NULL)
#22 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#23 src/XF/App.php(1931): XF\Mvc\Dispatcher->run()
#24 src/XF.php(328): XF\App->run()
#25 admin.php(13): XF::runApp('XF\\Admin\\App')
#26 {main}

-------------

Previous GuzzleHttp\Ring\Exception\ConnectException: cURL error 28: Operation timed out after 20001 milliseconds with 0 bytes received - src/vendor/guzzlehttp/ringphp/src/Client/CurlFactory.php:126
#0 src/vendor/guzzlehttp/ringphp/src/Client/CurlFactory.php(91): GuzzleHttp\Ring\Client\CurlFactory::createErrorResponse(Array, Array, Array)
#1 src/vendor/guzzlehttp/ringphp/src/Client/CurlHandler.php(96): GuzzleHttp\Ring\Client\CurlFactory::createResponse(Array, Array, Array, Array, Resource id #18)
#2 src/vendor/guzzlehttp/ringphp/src/Client/CurlHandler.php(68): GuzzleHttp\Ring\Client\CurlHandler->_invokeAsArray(Array)
#3 src/vendor/guzzlehttp/ringphp/src/Client/Middleware.php(30): GuzzleHttp\Ring\Client\CurlHandler->__invoke(Array)
#4 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(129): GuzzleHttp\Ring\Client\Middleware::GuzzleHttp\Ring\Client\{closure}(Array)
#5 src/vendor/guzzlehttp/guzzle/src/Client.php(165): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#6 src/addons/XFES/Elasticsearch/Api.php(351): GuzzleHttp\Client->send(Object(GuzzleHttp\Message\Request))
#7 src/addons/XFES/Elasticsearch/Api.php(403): XFES\Elasticsearch\Api->request('post', '_bulk', '{"index":{"_ind...', Array, Object(GuzzleHttp\Message\Request))
#8 src/addons/XFES/Elasticsearch/Api.php(171): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#9 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#10 src/addons/XFES/Search/Source/Elasticsearch.php(57): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#11 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#12 src/XF/Search/Search.php(59): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post))
#13 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('post', Object(XF\Mvc\Entity\ArrayCollection))
#14 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('post', 1495257, '2500')
#15 src/XF/Job/Manager.php(241): XF\Job\SearchRebuild->run(7.9999990463257)
#16 src/XF/Job/Manager.php(187): XF\Job\Manager->runJobInternal(Array, 7.9999990463257)
#17 src/XF/Job/Manager.php(103): XF\Job\Manager->runJobEntry(Array, 7.9999990463257)
#18 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#19 src/XF/Mvc/Dispatcher.php(249): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#20 src/XF/Mvc/Dispatcher.php(88): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', 'html', Object(XF\Mvc\ParameterBag), 'tools', Object(XF\Admin\Controller\Tools), NULL)
#21 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#22 src/XF/App.php(1931): XF\Mvc\Dispatcher->run()
#23 src/XF.php(328): XF\App->run()
#24 admin.php(13): XF::runApp('XF\\Admin\\App')
#25 {main}

 

[Xon]

That happens typically happens when ElasticSearch is stalling due to garbage collection due to low memory.