Search stops working after a while

[PrinceReborn]

Hi,

Hopefully somebody can help me out, I use enhanced search and have elasticsearch installed. And searching has been working fine but after a while I start getting the message "The search could not be completed. Please try again later." when trying to do searches. These are the error messages I am receiving in the server logs:

Error Info
XenForo_Exception: Elasticsearch error: SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[JoJzaSxeRHy0FPuAJQLoxA][mapleroy_xf][0]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1c9ec47b]}{[JoJzaSxeRHy0FPuAJQLoxA][mapleroy_xf][1]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1c64e23f]}{[JoJzaSxeRHy0FPuAJQLoxA][mapleroy_xf][2]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@72bf5413]}{[JoJzaSxeRHy0FPuAJQLoxA][mapleroy_xf][3]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1e51ffb6]}{[JoJzaSxeRHy0FPuAJQLoxA][mapleroy_xf][4]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@7a1fe4c6]}] - library/XenES/Search/SourceHandler/ElasticSearch.php:828
Generated By: Badohsai, 6 minutes ago
Stack Trace
#0 /home/mapleroy/public_html/forum/library/XenES/Search/SourceHandler/ElasticSearch.php(288): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
#1 /home/mapleroy/public_html/forum/library/XenForo/Search/SourceHandler/Abstract.php(115): XenES_Search_SourceHandler_ElasticSearch->executeSearch('scroll price gu...', false, Array, Array, false, '5000')
#2 /home/mapleroy/public_html/forum/library/XenForo/Search/Searcher.php(79): XenForo_Search_SourceHandler_Abstract->searchGeneral('scroll price gu...', Array, 'date', '5000')
#3 /home/mapleroy/public_html/forum/library/XenForo/ControllerPublic/Search.php(235): XenForo_Search_Searcher->searchGeneral('scroll price gu...', Array, 'date')
#4 /home/mapleroy/public_html/forum/library/XenES/Proxy/ControllerSearch.php(30): XenForo_ControllerPublic_Search->actionSearch()
#5 /home/mapleroy/public_html/forum/library/XenForo/FrontController.php(347): XenES_Proxy_ControllerSearch->actionSearch()
#6 /home/mapleroy/public_html/forum/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#7 /home/mapleroy/public_html/forum/index.php(13): XenForo_FrontController->run()
#8 {main}
Request State
array(3) {
["url"] => string(42) "http://mapleroyals.com/forum/search/search"
["_GET"] => array(0) {
}
["_POST"] => array(4) {
["keywords"] => string(18) "scroll price guide"
["users"] => string(0) ""
["date"] => string(0) ""
["_xfToken"] => string(8) "********"
}
}

I am able to fix the problem by rebuilding the index, but this happens again after about 12-24 hours or a seemingly random amount of time.

My elastic search setup:

Search options:

 

[Mike]

This likely relates to your Elasticsearch threadpool settings, which (from what I've read) seems to be changed from the default. A couple references:

http://elasticsearch-users.115913.n...-execution-queue-capacity-1000-td4061509.html
http://stackoverflow.com/questions/20683440/elasticsearch-gives-error-about-queue-size

There's a comment relating to ulimit in one of those, but mostly the comments seem to relate to the threadpool being small for your specific configuration. (Which may vary based on shards/copies, number of indexes on the server, etc.)

 

[vinafix]

difficult

I do not

 

[PrinceReborn]

This likely relates to your Elasticsearch threadpool settings, which (from what I've read) seems to be changed from the default. A couple references:

http://elasticsearch-users.115913.n...-execution-queue-capacity-1000-td4061509.html
http://stackoverflow.com/questions/20683440/elasticsearch-gives-error-about-queue-size

There's a comment relating to ulimit in one of those, but mostly the comments seem to relate to the threadpool being small for your specific configuration. (Which may vary based on shards/copies, number of indexes on the server, etc.)

difficult

I do not

The issue was resolved by upgrading to the latest ElasticSearch. It seems that a shellshock vulnerability was causing this issue and was potentially using the server as a zombie to DDoS. Thanks for the reply though. I urge anyone using 1.4.2 or lower to upgrade to the latest 1.4.5+