Fixed Result window too large with Elasticsearch 2.1.1 and ES 1.1.3

kontrabass

kontrabass

Well-known member
Got several of these errors last night after upgrading elasticsearch from 1.7 to 2.1.1, and XFES from 1.1.1 to 1.1.3, but today no errors so far....

Anyway, Xon suggested posting this as a bug report:

Code: 
XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [24000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":"tbsearch","node":"3Ta4Eu3qTTaC3s9-9Ibf7w","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [24000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}}]} - library/XenES/Search/SourceHandler/ElasticSearch.php:872
Generated By: NeilGB, Today at 12:58 AM
Stack Trace
#0 /home/talkbass/html/library/XenES/Search/SourceHandler/ElasticSearch.php(326): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
#1 /home/talkbass/html/library/SV/SearchImprovements/XenES/Search/SourceHandler/ElasticSearch.php(11): XenES_Search_SourceHandler_ElasticSearch->executeSearch('**', false, Array, Array, 'thread', '1500', Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post))
#2 /home/talkbass/html/library/XenForo/Search/SourceHandler/Abstract.php(152): SV_SearchImprovements_XenES_Search_SourceHandler_ElasticSearch->executeSearch('**', false, Array, Array, 'thread', '1500', Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post))
#3 /home/talkbass/html/library/XenForo/Search/Searcher.php(111): XenForo_Search_SourceHandler_Abstract->searchType(Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post), '**', Array, 'date', '1', '1500')
#4 /home/talkbass/html/library/XenForo/ControllerPublic/Search.php(240): XenForo_Search_Searcher->searchType(Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post), '**', Array, 'date', '1')
#5 /home/talkbass/html/library/Waindigo/Search/Extend/XenForo/ControllerPublic/Search.php(83): XenForo_ControllerPublic_Search->actionSearch()
#6 /home/talkbass/html/library/XenES/Proxy/ControllerSearch.php(30): Waindigo_Search_Extend_XenForo_ControllerPublic_Search->actionSearch()
#7 /home/talkbass/html/library/SV/SearchImprovements/XenForo/ControllerPublic/Search.php(41): XenES_Proxy_ControllerSearch->actionSearch()
#8 /home/talkbass/html/library/XenForo/FrontController.php(351): SV_SearchImprovements_XenForo_ControllerPublic_Search->actionSearch()
#9 /home/talkbass/html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#10 /home/talkbass/html/index.php(13): XenForo_FrontController->run()
#11 {main}
Request State
array(3) {
  ["url"] => string(37) "http://www.talkbass.com/search/search"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(8) {
    ["keywords"] => string(0) ""
    ["users"] => string(0) ""
    ["date"] => string(0) ""
    ["nodes"] => array(1) {
      [0] => string(3) "126"
    }
    ["type"] => array(1) {
      ["post"] => array(1) {
        ["group_discussion"] => string(1) "1"
      }
    }
    ["custom_fields"] => array(2) {
      ["strings"] => string(4) "five"
      ["state"] => string(0) ""
    }
    ["custom_fields_shown"] => array(4) {
      [0] => string(7) "strings"
      [1] => string(8) "fretless"
      [2] => string(5) "lefty"
      [3] => string(5) "state"
    }
    ["_xfToken"] => string(8) "********"
  }
}


I have XF set to return 1500 results, will try decreasing to 1000 results.
 
This is made much more common by the custom thread field search on your side. That appears to be implemented in such a way that it has to be done in SQL, so we "overfetch" to account for this (later filtering down). Similarly, when grouping by thread, we overfetch and filter down. Each multiplies the number of documents fetched by 4, but in combination, that ends up meaning by 16. Out of the box this can happen, but I don't think it would be anywhere near 20 times per day, even on a big board.

Regardless, I have set up a cap at fetching 10000 results. This is the out of the box limit for ES now, and fetching more than that to filter is probably unnecessary.

You can workaround this by adding:
Code: 
$dsl['size'] = min(10000, $dsl['size']);
Before:
Code: 
$response = XenES_Api::search($this->_indexName, $dsl);
In library/XenES/Search/SourceHanderl/ElasticSearch.php.

(There are two instances of the original line. Technically it only applies to the first one, though it shouldn't hurt to apply it to the second.)
 
Mike said:
This is made much more common by the custom thread field search on your side. That appears to be implemented in such a way that it has to be done in SQL, so we "overfetch" to account for this (later filtering down). Similarly, when grouping by thread, we overfetch and filter down. Each multiplies the number of documents fetched by 4, but in combination, that ends up meaning by 16. Out of the box this can happen, but I don't think it would be anywhere near 20 times per day, even on a big board.

Regardless, I have set up a cap at fetching 10000 results. This is the out of the box limit for ES now, and fetching more than that to filter is probably unnecessary.

You can workaround this by adding:
Code: 
$dsl['size'] = min(10000, $dsl['size']);
Before:
Code: 
$response = XenES_Api::search($this->_indexName, $dsl);
In library/XenES/Search/SourceHanderl/ElasticSearch.php.

(There are two instances of the original line. Technically it only applies to the first one, though it shouldn't hurt to apply it to the second.)
Click to expand...

Thanks Mike! :)
 
Back
Top Bottom