XF 1.5 Enhanced search 1.1.6 problems

Stuart Wright

Stuart Wright

Well-known member
I upgraded the forum to 1.5.11 and enhanced search to 1.1.6 from the previous versions this morning at about 10am.
In the last hour (so 7 hours later) all of a sudden we can't open threads or we get an error:
XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"},{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@21bf6c99 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818238]]"}},{"shard":1,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ca31432 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818265]]"}},{"shard":2,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@10f55c1a on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818294]]"}},{"shard":3,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3ac565a4 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818318]]"}},{"shard":4,"index":"xfavforums","node":"g4T7VLxDTV--dQJNvBReVQ","reason":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$4@3d7b3582 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f5a759e[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 764818356]]"}}]} - library/XenES/Search/SourceHandler/ElasticSearch.php:1006
Generated By: stopher, 17 minutes ago
Click to expand...
#0 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(361): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
#1 /home/sites/avforums/public_html/library/Editorial/Search/SourceHandler/ElasticSearch.php(163): XenES_Search_SourceHandler_ElasticSearch->executeSearch('Multi OR LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#2 /home/sites/avforums/public_html/library/AVForums/ThreadProduct/Search/SourceHandler/ElasticSearch.php(228): Editorial_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#3 /home/sites/avforums/public_html/library/XenForo/Search/SourceHandler/Abstract.php(149): AVForums_ThreadProduct_Search_SourceHandler_ElasticSearch->executeSearch('Multi |LNB', true, Array, Array, 'thread', 5, Object(XenForo_Search_DataHandler_Post))
#4 /home/sites/avforums/public_html/library/XenForo/Search/Searcher.php(109): XenForo_Search_SourceHandler_Abstract->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#5 /home/sites/avforums/public_html/library/XMWidgets/Threads/Similar.php(64): XenForo_Search_Searcher->searchType(Object(XenForo_Search_DataHandler_Post), 'Multi |LNB', Array, 'relevance', 1, 5)
#6 /home/sites/avforums/public_html/library/WidgetFramework/WidgetRenderer.php(683): XMWidgets_Threads_Similar->_render(Array, 'thread_view', Array, Object(XenForo_Template_Public))
#7 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(397): WidgetFramework_WidgetRenderer->render(Array, 'thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#8 /home/sites/avforums/public_html/library/WidgetFramework/Core.php(316): WidgetFramework_Core->_renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Object(_WidgetFramework_ArrayOfString))
#9 /home/sites/avforums/public_html/library/WidgetFramework/Listener.php(126): WidgetFramework_Core->renderWidgetsFor('thread_view', Array, Object(XenForo_Template_Public), Array)
#10 /home/sites/avforums/public_html/library/XenForo/CodeEvent.php(58): WidgetFramework_Listener::template_post_render('thread_view', '\n\n\t\n\t\n\n\n\n\n\n\n\n\n\n...', Array, Object(XenForo_Template_Public))
#11 /home/sites/avforums/public_html/library/XenForo/Template/Abstract.php(195): XenForo_CodeEvent::fire('template_post_r...', Array, 'thread_view')
#12 /home/sites/avforums/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#13 /home/sites/avforums/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(124): XenForo_Template_Public->render()
#14 /home/sites/avforums/public_html/library/XenForo/FrontController.php(639): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#15 /home/sites/avforums/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#16 /home/sites/avforums/public_html/index.php(13): XenForo_FrontController->run()
#17 {main}
Request State
array(3) {
["url"] => string(51) "https://www.avforums.com/threads/multi-lnb.2060803/"
["_GET"] => array(1) {
["/threads/multi-lnb_2060803/"] => string(0) ""
}
["_POST"] => array(0) {
}
}
Click to expand...
I have had to disabled enhanced search so people can get to threads but our editorial system assumes Enhanced Search is running so we're getting lots of server errors every second.
Is there any help with regard to what changed in 1.1.6?
Thanks
 
Sort by date Sort by votes
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
 
Upvote 0 Downvote
Did you just upgrade then? There are notes about index compatibility, so if your index was created with Elasticsearch 1 (I believe), you likely need to reindex (might even need the index to be deleted).
 
Upvote 0 Downvote
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
 
Upvote 0 Downvote
Chris D said:
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
Click to expand...
If I do that, I'll have to uninstall Enhanced Search and reinstall and reindex everything, right?
The odd thing is that AVForums was fine for 7 hours.
 
Upvote 0 Downvote
Mike said:
The changes really only apply if you're running Elasticsearch 5, so I think it's just coincidental. It looks to me like Elasticsearch's queue is full, so it's rejecting searches. That probably has to be investigated more internal to ES. It might correspond to a load spike on the server for example.
Click to expand...
Ok, I was wrong about Elastic.

"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.0.2",
"build_timestamp" : "2015-12-16T12:49:58Z",
"build_snapshot" : false,
"lucene_version" : "5.2.1"
Click to expand...
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.
 
Upvote 0 Downvote
It may have just been a glitch in Elasticsearch then. Maybe it was even another process that caused a backlog in ES. It's hard to say. In terms of normal searches, I believe the code only changed if you're running ES 5.
 
Upvote 0 Downvote
Stuart Wright said:
There is actually no way to know whether we had a spike or not until Google Analytics rolls over to tomorrow. And even then, it would have to be a pretty big spike to register.
Click to expand...

Do you have Munin installed? I can't remember if that was included by Nimbus or if we asked them specifically for it but it's what I usually use for monitoring of spikes, etc.
 
Upvote 0 Downvote
Chris D said:
Thinking about that particular add on, though, it might not be a bad idea to revert back to ES1/2 until you've had a chance to test that in non live conditions.
Click to expand...
So hoping it was just a glitch, I've re-enabled the product associator addon and the forum is fine.
Odd. We've not had an issue like that with Elastic Search as long as I can remember.
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

M
  • Question Question
ES 2.2 Can Enhanced Search treat & or AND the same - Fuzzy search?
Replies
0
Views
33
Mikethemix
M
Mouth
  • Question Question
XF 2.2 No search results after turning off enhanced, falling back to inbuilt, and rebuilding search
Replies
2
Views
48
Mouth
Mouth
Wildcat Media
  • Question Question
Enhanced Search 1.1.8 -- maximum ElasticSearch version?
Replies
11
Views
2K
Wildcat Media
Wildcat Media
bloop
  • Question Question
Xenforo Enhanced Search 1.1.4 - Error Message
Replies
1
Views
802
Overscan
Overscan
TAIFUN
Not a bug XenForo Enhanced Search 2.2.2 unknown query [type] src/addons/XFES/Elasticsearch/Api.php:424
Replies
2
Views
920
PaulB
PaulB
Back
Top Bottom