XF 2.2 XF 2.28 Patch with Elastic search only returning 400 pages

[ebikes.com]

Hi there, having a serious problem with our forum.
Some members have post counts in the 30,000+ range and in those cases we can only return 400 pages of those posts, or 10,000 of their posts.
For certain users this can clip off >75% of their posting history when you search for all posts under their username.

We have adequate RAM on the server and no time outs, out of disk warnings, php warnings etc during these searches.

Here are our search options for Xenforo ( enhanced search is all set to defaults and also, our ES is bone stock as well )



Elastic Search doesn't seem to be taking the above setting into account. How can i make elasticsearch look deeper than 400 pages ( 10,000 posts ) when searching through a user's history?

 

[ebikes.com]

Turns out this 400 page limit correlates with a maximum of 10000 find results ( 25 x 400 = 10000 )
Setting it to 15000 still returns 400 pages.

Without elastic search on on our development copy, setting the maximum results to 25000 gives 1000 pages as expected.

How can i remove this artificial limitation of enhanced search? we have the horsepower to handle the size, and an incredible amount of content.

 

[Jeremy P]

It's not exactly an artificial limitation. Elasticsearch will run into performance issues beyond 10,000 results, and the search system does not offer cursor-based pagination (ie. search_after). Further, things like the find-new system do not use Elasticsearch at all and a limit that high is liable to cause performance issues with MySQL. My only recommendation is to narrow your search using the newer than/older than filters when necessary. I'm not sure how much utility there really is in an exhaustive listing, but the search system was not really designed to accomodate this.

 

[ebikes.com]

Thanks for the quick response.

I understand the resource usage implications but again we have a lot of power, and we have senior PHP developers and a linux system administrator to help troubleshoot performance based issues. We have not gone so far as to try to patch code but we would be willing to patch it and maintain that patch indefinetely.

MySQL search performance for large searches is not great - however it will return the full result set when asked. What it won't do is search for 3 character words even if you set it to do that. We bought the elastic search plugin hoping it would help the 3 character word issue, but this is a show stopper.

Between the two options we have less search functionality than what we had on PHPBB

Being able to search this far is actually critical to our forum; we've been around since 2006 and have a lot of content. A lot of the older content is useful and may need to be referenced much later. We have 11 power users who have >10,000 posts and at first glance a user would get a wrong impression on how much they posted. We also have many common terms that exceed this artificial limit, preventing our users from finding old posts.

What can be done to improve this?

 

[Jeremy P]

You would need to edit the Elasticsearch configuration to change index.max_result_window and edit the XFES\Search\Source\Elasticsearch::getSearchSizeDsl method to match it, at a minimum.

If you do start running into performance issues, you'd need an add-on that implements cursor-based pagination for XFES. I'm not aware of any existing ones, and it's likely to be a complicated undertaking.

 

[ebikes.com]

Thank you so much. I'm glad this isn't an issue with the Xenforo software itself.

If this results in significantly increased memory usage ( undesirable ), is it possible to revert to the mysql engine but force it to search for down to 3 character terms?

I know this would probably result in a gigantic XF search index; however; disk is significantly cheaper than the 11gb of ram that elastic search reserves, so i wouldn't mind.

 

[Jeremy P]

I believe you'd have to configure MySQL to support it, change the minimum length in the control panel, and rebuild the search index.

 

[Paul B]

If this results in significantly increased memory usage ( undesirable ), is it possible to revert to the mysql engine but force it to search for down to 3 character terms?

Disable XFES, set the minimum word length in the ACP, edit my.cnf to match, rebuild the search index.

 

[ebikes.com]

Thanks guys for the great support, i'm sorry i came in here with a crabby attitude.

What's the my.cnf setting for that?
A quick stack overflow search tells me that this would work for innodb tables:
[mysqld]
innodb_ft_min_token_size=2
ft_min_word_len=2

But from what i see xenforo has a mixture of table types so i am not sure if this is the silver bullet.

We run a fairly bone stock mysql 8.0.x configuration with a really big innodb cache as the only real modification.