XF 1.5 Enhanced search 1.1.6 problems

Stuart Wright

Well-known member
I upgraded the forum to 1.5.11 and enhanced search to 1.1.6 from the previous versions this morning at about 10am.
In the last hour (so 7 hours later) all of a sudden we can't open threads or we get an error:
XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"}},{"shard":1,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"}},{"shard":2,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"}},{"shard":3,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"}},{"shard":4,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}}]} - library/XenES/Search/SourceHandler/ElasticSearch.php:1006
Generated By: stopher, 17 minutes ago
#0 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(361): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
#1 /home/sites/avforums/public_html/library/Editorial/Search/SourceHandler/ElasticSearch.php(163): XenES_Search_SourceHandler_ElasticSearch->executeSearch('Multi OR LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#2 /home/sites/avforums/public_html/library/AVForums/ThreadProduct/Search/SourceHandler/ElasticSearch.php(228): Editorial_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#3 /home/sites/avforums/public_html/library/XenForo/Search/SourceHandler/Abstract.php(149): AVForums_ThreadProduct_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#4 /home/sites/avforums/public_html/library/XenForo/Search/Searcher.php(109): XenForo_Search_SourceHandler_Abstract->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#5 /home/sites/avforums/public_html/library/XMWidgets/Threads/Similar.php(64): XenForo_Search_Searcher->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#6 /home/sites/avforums/public_html/library/WidgetFramework/WidgetRenderer.php(683): XMWidgets_Threads_Similar->_render(Array, 'thread_view', Array, Object(XenForo_Template_Public))
#7 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(397): WidgetFramework_WidgetRenderer->render(Array, 'thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#8 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(316): WidgetFramework_Core->_renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#9 /home/sites/avforums/public_html/library/WidgetFramework/Listener.php(126): WidgetFramework_Core->renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Array)
#10 /home/sites/avforums/public_html/library/XenForo/CodeEvent.php(58): WidgetFramework_Listener::template_post_render('thread_view', '\n\n\t\n\t\n\n\n\n\n\n\n\n\n\n...', Array, Object(XenForo_Template_Public))
#11 /home/sites/avforums/public_html/library/XenForo/Template/Abstract.php(195): XenForo_CodeEvent::fire('template_post_r...', Array, 'thread_view')
#12 /home/sites/avforums/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#13 /home/sites/avforums/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(124): XenForo_Template_Public->render()
#14 /home/sites/avforums/public_html/library/XenForo/FrontController.php(639): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#15 /home/sites/avforums/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#16 /home/sites/avforums/public_html/index.php(13): XenForo_FrontController->run()
#17 {main}
Request State
array(3) {
["url"] => string(51) "https://www.avforums.com/threads/multi-lnb.2060803/"
["_GET"] => array(1) {
["/threads/multi-lnb_2060803/"] => string(0) ""
}
["_POST"] => array(0) {
}
}
I have had to disabled enhanced search so people can get to threads but our editorial system assumes Enhanced Search is running so we're getting lots of server errors every second.
Is there any help with regard to what changed in 1.1.6?
Thanks
 
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
 
Did you just upgrade then? There are notes about index compatibility, so if your index was created with Elasticsearch 1 (I believe), you likely need to reindex (might even need the index to be deleted).
 
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
 
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
If I do that, I'll have to uninstall Enhanced Search and reinstall and reindex everything, right?
The odd thing is that AVForums was fine for 7 hours.
 
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
Ok, I was wrong about Elastic.

"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.0.2",
"build_timestamp" : "2015-12-16T12:49:58Z",
"build_snapshot" : false,
"lucene_version" : "5.2.1"
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.
 
It may have just been a glitch in Elasticsearch then. Maybe it was even another process that caused a backlog in ES. It's hard to say. In terms of normal searches, I believe the code only changed if you're running ES 5.
 
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.

Do you have Munin installed? I can't remember if that was included by Nimbus or if we asked them specifically for it but it's what I usually use for monitoring of spikes, etc.
 
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
So hoping it was just a glitch, I've re-enabled the product associator addon and the forum is fine.
Odd. We've not had an issue like that with Elastic Search as long as I can remember.
 
Top Bottom