Elasticsearch indexing error: Failed to Connect: Connection refused

[rfc0001]

I installed Elasticsearch 2.x and was running with no issues for several days.

Then I uninstalled Elasticsearch 2.x and installed Elasticsearch 6.x, and for the following two days I've gotten the following server error in the morning:

Elasticsearch indexing error (queued): cURL error 7: Failed to connect to localhost port 9200: Connection refused

Stack trace
#0 src/addons/XFES/Elasticsearch/Api.php(90): XFES\Elasticsearch\Api->request('get', '/')
#1 src/addons/XFES/Elasticsearch/Api.php(107): XFES\Elasticsearch\Api->version()
#2 src/addons/XFES/Elasticsearch/Api.php(134): XFES\Elasticsearch\Api->majorVersion()
#3 src/addons/XFES/Elasticsearch/Api.php(317): XFES\Elasticsearch\Api->isTypelessIndex()
#4 src/addons/XFES/Elasticsearch/Api.php(152): XFES\Elasticsearch\Api->requestById('put', 'post', 47676, Array)
#5 src/addons/XFES/Search/Source/Elasticsearch.php(66): XFES\Elasticsearch\Api->index('post', 47676, Array)
#6 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#7 src/XF/Behavior/Indexable.php(46): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post), false)
#8 src/XF.php(244): XF\Behavior\Indexable->XF\Behavior\{closure}()
#9 src/XF/Mvc/Dispatcher.php(137): XF::triggerRunOnce(true)
#10 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#11 src/XF/App.php(1880): XF\Mvc\Dispatcher->run()
#12 src/XF.php(328): XF\App->run()
#13 index.php(13): XF::runApp('XF\\Pub\\App')
#14 {main}

-------------

Previous GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 9200: Connection refused - src/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:49
#0 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(103): GuzzleHttp\Exception\RequestException::wrapException(Object(GuzzleHttp\Message\Request), Object(GuzzleHttp\Ring\Exception\ConnectException))
#1 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(132): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#2 src/vendor/react/promise/src/FulfilledPromise.php(25): GuzzleHttp\RequestFsm->GuzzleHttp\{closure}(Array)
#3 src/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
#4 src/vendor/guzzlehttp/guzzle/src/Message/FutureResponse.php(43): GuzzleHttp\Ring\Future\CompletedFutureValue->then(Object(Closure), NULL, NULL)
#5 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(134): GuzzleHttp\Message\FutureResponse::proxy(Object(GuzzleHttp\Ring\Future\CompletedFutureArray), Object(Closure))
#6 src/vendor/guzzlehttp/guzzle/src/Client.php(165): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#7 src/addons/XFES/Elasticsearch/Api.php(351): GuzzleHttp\Client->send(Object(GuzzleHttp\Message\Request))
#8 src/addons/XFES/Elasticsearch/Api.php(90): XFES\Elasticsearch\Api->request('get', '/')
#9 src/addons/XFES/Elasticsearch/Api.php(107): XFES\Elasticsearch\Api->version()
#10 src/addons/XFES/Elasticsearch/Api.php(134): XFES\Elasticsearch\Api->majorVersion()
#11 src/addons/XFES/Elasticsearch/Api.php(317): XFES\Elasticsearch\Api->isTypelessIndex()
#12 src/addons/XFES/Elasticsearch/Api.php(152): XFES\Elasticsearch\Api->requestById('put', 'post', 47676, Array)
#13 src/addons/XFES/Search/Source/Elasticsearch.php(66): XFES\Elasticsearch\Api->index('post', 47676, Array)
#14 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#15 src/XF/Behavior/Indexable.php(46): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post), false)
#16 src/XF.php(244): XF\Behavior\Indexable->XF\Behavior\{closure}()
#17 src/XF/Mvc/Dispatcher.php(137): XF::triggerRunOnce(true)
#18 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#19 src/XF/App.php(1880): XF\Mvc\Dispatcher->run()
#20 src/XF.php(328): XF\App->run()
#21 index.php(13): XF::runApp('XF\\Pub\\App')
#22 {main}

-------------

Previous GuzzleHttp\Ring\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 9200: Connection refused - src/vendor/guzzlehttp/ringphp/src/Client/CurlFactory.php:126
#0 src/vendor/guzzlehttp/ringphp/src/Client/CurlFactory.php(91): GuzzleHttp\Ring\Client\CurlFactory::createErrorResponse(Array, Array, Array)
#1 src/vendor/guzzlehttp/ringphp/src/Client/CurlHandler.php(96): GuzzleHttp\Ring\Client\CurlFactory::createResponse(Array, Array, Array, Array, Resource id #264)
#2 src/vendor/guzzlehttp/ringphp/src/Client/CurlHandler.php(68): GuzzleHttp\Ring\Client\CurlHandler->_invokeAsArray(Array)
#3 src/vendor/guzzlehttp/ringphp/src/Client/Middleware.php(30): GuzzleHttp\Ring\Client\CurlHandler->__invoke(Array)
#4 src/vendor/guzzlehttp/guzzle/src/RequestFsm.php(129): GuzzleHttp\Ring\Client\Middleware::GuzzleHttp\Ring\Client\{closure}(Array)
#5 src/vendor/guzzlehttp/guzzle/src/Client.php(165): GuzzleHttp\RequestFsm->__invoke(Object(GuzzleHttp\Transaction))
#6 src/addons/XFES/Elasticsearch/Api.php(351): GuzzleHttp\Client->send(Object(GuzzleHttp\Message\Request))
#7 src/addons/XFES/Elasticsearch/Api.php(90): XFES\Elasticsearch\Api->request('get', '/')
#8 src/addons/XFES/Elasticsearch/Api.php(107): XFES\Elasticsearch\Api->version()
#9 src/addons/XFES/Elasticsearch/Api.php(134): XFES\Elasticsearch\Api->majorVersion()
#10 src/addons/XFES/Elasticsearch/Api.php(317): XFES\Elasticsearch\Api->isTypelessIndex()
#11 src/addons/XFES/Elasticsearch/Api.php(152): XFES\Elasticsearch\Api->requestById('put', 'post', 47676, Array)
#12 src/addons/XFES/Search/Source/Elasticsearch.php(66): XFES\Elasticsearch\Api->index('post', 47676, Array)
#13 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#14 src/XF/Behavior/Indexable.php(46): XF\Search\Search->index('post', Object(XFMG\XF\Entity\Post), false)
#15 src/XF.php(244): XF\Behavior\Indexable->XF\Behavior\{closure}()
#16 src/XF/Mvc/Dispatcher.php(137): XF::triggerRunOnce(true)
#17 src/XF/Mvc/Dispatcher.php(41): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#18 src/XF/App.php(1880): XF\Mvc\Dispatcher->run()
#19 src/XF.php(328): XF\App->run()
#20 index.php(13): XF::runApp('XF\\Pub\\App')
#21 {main}

Today, this error occured at 7:10AM. There are a # of default Cron jobs that run at 7:10AM:

1. Rebuild member stats cache
2. XFMG: Cache gallery statistics
3. Hourly clean up
4. Perform file health check

If I check the service status (sudo service elasticsearch status) I get:
elasticsearch dead but pid file exists

If I restart the service shutting down the services FAILED but starting it is OK

Checking all the Elasticsearch logs doesn't show any failures of any kind.

I searched the "elastic search dead but pid file exists" error and only found someone who got this error due to bing out of disk space (which he got an exception stating this was the error when he restarted). This is not my issue.

I'm running a default install of Elasticsearch, other than I went through the recommended configurations and increased threads to 4096, locked memory (1GB default), etc. I couldn't make a couple of the recommended changes due to being on a VPS. I'm wondering if I should reinstall without any custom configurations?

Any other ideas? Thanks.

 

[rfc0001]

Here are the recommended config changes I was running in /etc/security/limits.conf:

elasticsearch    -       nproc           4096
elasticsearch    -       nofile          65536
elasticsearch    -       memlock         unlimited

And settings the following config settings in Elasticsearch config:

• cluster.name
• node.name
• bootstrap.memory_lock
• network.host
• http.port

The only recommended setting I couldn't set was mmap count limit, which I don't have permissions to modify since I'm on a VPS:
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/vm-max-map-count.html

 

[rfc0001]

Uninstalled/reinstalled and didn't change any settings other than cluster.name, node.name, network.host, and http.port and no issues today. Will run a couple days without issues then re-apply recommended settings in limits.conf and see if it blows up again. My guess is increasing threads without increasing mmap limit caused problems.