XF 1.5 Enhanced search 1.1.6 problems

Stuart Wright

Well-known member
I upgraded the forum to 1.5.11 and enhanced search to 1.1.6 from the previous versions this morning at about 10am.
In the last hour (so 7 hours later) all of a sudden we can't open threads or we get an error:
XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"}},{"shard":1,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"}},{"shard":2,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"}},{"shard":3,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"}},{"shard":4,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}}]} - library/XenES/Search/SourceHandler/ElasticSearch.php:1006
Generated By: stopher, 17 minutes ago
#0 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(361): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
#1 /home/sites/avforums/public_html/library/Editorial/Search/SourceHandler/ElasticSearch.php(163): XenES_Search_SourceHandler_ElasticSearch->executeSearch('Multi OR LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#2 /home/sites/avforums/public_html/library/AVForums/ThreadProduct/Search/SourceHandler/ElasticSearch.php(228): Editorial_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#3 /home/sites/avforums/public_html/library/XenForo/Search/SourceHandler/Abstract.php(149): AVForums_ThreadProduct_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#4 /home/sites/avforums/public_html/library/XenForo/Search/Searcher.php(109): XenForo_Search_SourceHandler_Abstract->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#5 /home/sites/avforums/public_html/library/XMWidgets/Threads/Similar.php(64): XenForo_Search_Searcher->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#6 /home/sites/avforums/public_html/library/WidgetFramework/WidgetRenderer.php(683): XMWidgets_Threads_Similar->_render(Array, 'thread_view', Array, Object(XenForo_Template_Public))
#7 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(397): WidgetFramework_WidgetRenderer->render(Array, 'thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#8 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(316): WidgetFramework_Core->_renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#9 /home/sites/avforums/public_html/library/WidgetFramework/Listener.php(126): WidgetFramework_Core->renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Array)
#10 /home/sites/avforums/public_html/library/XenForo/CodeEvent.php(58): WidgetFramework_Listener::template_post_render('thread_view', '\n\n\t\n\t\n\n\n\n\n\n\n\n\n\n...', Array, Object(XenForo_Template_Public))
#11 /home/sites/avforums/public_html/library/XenForo/Template/Abstract.php(195): XenForo_CodeEvent::fire('template_post_r...', Array, 'thread_view')
#12 /home/sites/avforums/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#13 /home/sites/avforums/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(124): XenForo_Template_Public->render()
#14 /home/sites/avforums/public_html/library/XenForo/FrontController.php(639): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#15 /home/sites/avforums/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#16 /home/sites/avforums/public_html/index.php(13): XenForo_FrontController->run()
#17 {main}
Request State
array(3) {
["url"] => string(51) "https://www.avforums.com/threads/multi-lnb.2060803/"
["_GET"] => array(1) {
["/threads/multi-lnb_2060803/"] => string(0) ""
}
["_POST"] => array(0) {
}
}
I have had to disabled enhanced search so people can get to threads but our editorial system assumes Enhanced Search is running so we're getting lots of server errors every second.
Is there any help with regard to what changed in 1.1.6?
Thanks
 

Chris D

XenForo developer
Staff member
That looks related to the Thread Product add-on.

If you disable that, and re-enable ES, does the problem go away?
 

Stuart Wright

Well-known member
Although it might coincide with an email I sent to Tim at Nimbus 4 minutes ago asking to restart Elastic. Not sure how to check whether Elastic is running or not.
 

Mike

XenForo developer
Staff member
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
 

Mike

XenForo developer
Staff member
Did you just upgrade then? There are notes about index compatibility, so if your index was created with Elasticsearch 1 (I believe), you likely need to reindex (might even need the index to be deleted).
 

Chris D

XenForo developer
Staff member
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
 

Stuart Wright

Well-known member
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
If I do that, I'll have to uninstall Enhanced Search and reinstall and reindex everything, right?
The odd thing is that AVForums was fine for 7 hours.
 

Stuart Wright

Well-known member
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
Ok, I was wrong about Elastic.

"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.0.2",
"build_timestamp" : "2015-12-16T12:49:58Z",
"build_snapshot" : false,
"lucene_version" : "5.2.1"
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.
 

Mike

XenForo developer
Staff member
It may have just been a glitch in Elasticsearch then. Maybe it was even another process that caused a backlog in ES. It's hard to say. In terms of normal searches, I believe the code only changed if you're running ES 5.
 

RobParker

Well-known member
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.
Do you have Munin installed? I can't remember if that was included by Nimbus or if we asked them specifically for it but it's what I usually use for monitoring of spikes, etc.
 

Stuart Wright

Well-known member
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
So hoping it was just a glitch, I've re-enabled the product associator addon and the forum is fine.
Odd. We've not had an issue like that with Elastic Search as long as I can remember.
 
Top