XF 1.4 Elasticsearch only displaying a handful of results

[Joe Link]

This morning I noticed that searching for a common term on my site yields only a handful of results. There should literally be hundreds if not thousands for this particular brand name, and it returned two. We have around 1.1M posts.

I rebuilt the search cache without deleting the index. I searched again and received three pages of results, still far below what I should expect. About five minutes later, after looking for an answer, I searched for the same term again. This time it returned the original two results. I have not tried rebuilding the search with the delete index option selected.

I believe the only search-related add-ons I have installed are Conversation Essentials, Showcase, and XenMedia Gallery.

Where should I start?

 

[Paul B]

If you are indexing conversation content then that is the reason why.

 

[Paul B]

To follow up, the last time we looked into this, this is what Mike had to say on it:

What you're running into is partially a technical limit, but significantly exacerbated by add-ons. The XF option to control the maximum results relates to the maximum number of results we receive from the search index pre-permission filtering. If a term appears 10,000 times, then we'll get the top 200 (say) results and then apply permission filtering on that. If you can't view any of those results, it will tell you no results, even if you could have viewed 201-400. You can increase the maximum number of results, but a limit is always going to have to be placed or there's potential for very bad performance (particularly if the search isn't fast).

This issue is actually one of the reasons why we haven't added conversation searching. In conversations, 99% of the data isn't accessible by other users. This means that 99% of the results will be thrown away. If people are talking about a common term in a conversation, it's going to mess with my results. Effectively, putting conversations in the search index mostly serves to push other results out.

You can try rebuilding the search index, in case it is related to that, but it's likely the same case.

 

[Joe Link]

Thanks for the replies, that's definitely what it is. I disabled the mod, rebuilt the search index, and it's returning 100 results every time.

What I don't understand is how a popular paid add-on could cripple such a huge feature. Maybe I don't have it configured right? @Daniel Hood

 

[Paul B]

The reason is as I posted above.

Search results are fetched first and then filtered based on permissions.
So if a lot of people are using the searched for term in their conversations, which you don't have access to, all of those fetched results are discarded.

 

[Mike]

It's really a conceptual issue with having content in the index that isn't viewable to everyone who searches. The same issue can happen if you have a forum with a huge amount of content that most people can't see. This isn't particularly common though, where as with conversations, it essentially happens by definition.

 

[Joe Link]

Sorry, to clarify, I get the technical reasons

 

[Daniel Hood]

Interesting. Sorry about that @Joe Link. I was already planning on adding a toggle to the next release of ConvEss to turn off conversation indexing.

 

[Joe Link]

Interesting. Sorry about that @Joe Link. I was already planning on adding a toggle to the next release of ConvEss to turn off conversation indexing.

I appreciate that Daniel, not a problem.

This should work until then? If so, is it reversible once you add the option?

Deleting the searchhandler from the xf_content_type_field table should work.

 

[Daniel Hood]

I appreciate that Daniel, not a problem.

This should work until then? If so, is it reversible once you add the option?

That's a pretty untested solution, in theory it should work. You might also have to disable a template modificaton for the search tabs.

 

[Joe Link]

That's a pretty untested solution, in theory it should work. You might also have to disable a template modificaton for the search tabs.

I'd rather not be a guinea pig then

Any idea when the next version will be released?

 

[Daniel Hood]

I'd rather not be a guinea pig then

Any idea when the next version will be released?

Tuesday/Wednesday most likely. I'm going away for the weekend and will have limited computer time. Have to finalize some stuff for my wedding.

 

[Joe Link]

Sounds good, congrats by the way!