1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Fixed Result window too large with Elasticsearch 2.1.1 and ES 1.1.3

Discussion in 'Enhanced Search Resolved Bugs' started by kontrabass, Dec 18, 2015.

  1. kontrabass

    kontrabass Well-Known Member

    Got several of these errors last night after upgrading elasticsearch from 1.7 to 2.1.1, and XFES from 1.1.1 to 1.1.3, but today no errors so far....

    Anyway, Xon suggested posting this as a bug report:

    XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [24000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":"tbsearch","node":"3Ta4Eu3qTTaC3s9-9Ibf7w","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [24000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter."}}]} - library/XenES/Search/SourceHandler/ElasticSearch.php:872
    Generated By: NeilGB, Today at 12:58 AM
    Stack Trace
    #0 /home/talkbass/html/library/XenES/Search/SourceHandler/ElasticSearch.php(326): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(Object(stdClass), true)
    #1 /home/talkbass/html/library/SV/SearchImprovements/XenES/Search/SourceHandler/ElasticSearch.php(11): XenES_Search_SourceHandler_ElasticSearch->executeSearch('**', false, Array, Array, 'thread', '1500', Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post))
    #2 /home/talkbass/html/library/XenForo/Search/SourceHandler/Abstract.php(152): SV_SearchImprovements_XenES_Search_SourceHandler_ElasticSearch->executeSearch('**', false, Array, Array, 'thread', '1500', Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post))
    #3 /home/talkbass/html/library/XenForo/Search/Searcher.php(111): XenForo_Search_SourceHandler_Abstract->searchType(Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post), '**', Array, 'date', '1', '1500')
    #4 /home/talkbass/html/library/XenForo/ControllerPublic/Search.php(240): XenForo_Search_Searcher->searchType(Object(Waindigo_Search_Extend_XenForo_Search_DataHandler_Post), '**', Array, 'date', '1')
    #5 /home/talkbass/html/library/Waindigo/Search/Extend/XenForo/ControllerPublic/Search.php(83): XenForo_ControllerPublic_Search->actionSearch()
    #6 /home/talkbass/html/library/XenES/Proxy/ControllerSearch.php(30): Waindigo_Search_Extend_XenForo_ControllerPublic_Search->actionSearch()
    #7 /home/talkbass/html/library/SV/SearchImprovements/XenForo/ControllerPublic/Search.php(41): XenES_Proxy_ControllerSearch->actionSearch()
    #8 /home/talkbass/html/library/XenForo/FrontController.php(351): SV_SearchImprovements_XenForo_ControllerPublic_Search->actionSearch()
    #9 /home/talkbass/html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #10 /home/talkbass/html/index.php(13): XenForo_FrontController->run()
    #11 {main}
    Request State
    array(3) {
      ["url"] => string(37) "http://www.talkbass.com/search/search"
      ["_GET"] => array(0) {
      ["_POST"] => array(8) {
        ["keywords"] => string(0) ""
        ["users"] => string(0) ""
        ["date"] => string(0) ""
        ["nodes"] => array(1) {
          [0] => string(3) "126"
        ["type"] => array(1) {
          ["post"] => array(1) {
            ["group_discussion"] => string(1) "1"
        ["custom_fields"] => array(2) {
          ["strings"] => string(4) "five"
          ["state"] => string(0) ""
        ["custom_fields_shown"] => array(4) {
          [0] => string(7) "strings"
          [1] => string(8) "fretless"
          [2] => string(5) "lefty"
          [3] => string(5) "state"
        ["_xfToken"] => string(8) "********"

    I have XF set to return 1500 results, will try decreasing to 1000 results.
    Xon likes this.
  2. BamaStangGuy

    BamaStangGuy Well-Known Member

    Getting same
  3. kontrabass

    kontrabass Well-Known Member

    Get about 20 of these errors per day. Otherwise search seems to work fine...
  4. Mike

    Mike XenForo Developer Staff Member

    This is made much more common by the custom thread field search on your side. That appears to be implemented in such a way that it has to be done in SQL, so we "overfetch" to account for this (later filtering down). Similarly, when grouping by thread, we overfetch and filter down. Each multiplies the number of documents fetched by 4, but in combination, that ends up meaning by 16. Out of the box this can happen, but I don't think it would be anywhere near 20 times per day, even on a big board.

    Regardless, I have set up a cap at fetching 10000 results. This is the out of the box limit for ES now, and fetching more than that to filter is probably unnecessary.

    You can workaround this by adding:
    $dsl['size'] = min(10000, $dsl['size']);
    $response = XenES_Api::search($this->_indexName, $dsl);
    In library/XenES/Search/SourceHanderl/ElasticSearch.php.

    (There are two instances of the original line. Technically it only applies to the first one, though it shouldn't hurt to apply it to the second.)
    Mert, Rudy and kontrabass like this.
  5. kontrabass

    kontrabass Well-Known Member

    Thanks Mike! :)

Share This Page