Some searches returning fewer than usual results

[Parsnip]

I've installed Enhanced Search and mostly it appears to be working well. However I have some situations where it returns drastically less results that the standard search.

For example searching for 'parts' with Elasticsearch disabled returns 73 results, with the keyword found in the body or title. But with it enabled it returns only a single result.

I have the default search order set to Relevance. When I change the order to Most Recent I see a lot of results again with 82 being returned.

Deleting the search index and rebuilding has not helped.

Thanks.

 

[rdn]

I can confirm this behavior with the same search keyword on my own forum .

 

[Mike]

Do you have any add-ons that affect the content that is being indexed? Notably, one that indexes conversations?

 

[rdn]

Do you have any add-ons that affect the content that is being indexed? Notably, one that indexes conversations?

Me, yes addon from @Xon

 

[Parsnip]

Yes but this testing user does not have any conversation messages and disabling the Conversation Essentials add-on doesn't change the results returned.

 

[Xon]

The problem is that the conversations aren't filtered out before XenForo sees them, this is a known design flaw where the general search isn't pushing down per-content type information to restrict the query.

If you search just posts & threads, or the quick search on a forum, you'll get a vastly better search experience as it isn't going through the general search which has a number of technical limitations.

 

[Parsnip]

The user I am testing with is clean and has never had any conversation messages. The discrepancies I'm seeing in the results relate only to posts and threads.

 

[Xon]

The user I am testing with is clean and has never had any conversation messages. The discrepancies I'm seeing in the results relate only to posts and threads.

It matters how you are doing the search. Are you using the general search? Or posts and threads?

That the user has never had any conversation messages doesn't matter, since what matters is there is any conversation messages!

 

[Parsnip]

I don't quote follow you with regards to the cause sorry, but I've been using the quick search box at the top right. Why would the conversations need to be filtered out? I would prefer that if there were results then they were included.

So going into Search Threads and Posts and setting the Order By there, I am seeing more results returned for Relevance (195) than Most Recent (143) now.

Is there a fix or patch for this?

 

[Parsnip]

Ok, so I've spent a few hours on this now and still aren't much wiser as to what the exact problem is even after reading up here more. Something about conversation messages pushing posts/threads out of the search results.

Does the problem lie with Xenforo or the add-on? What are my options for resolving it?

Right now, after all my efforts, I seem to have given my users a worse search experience. I can only think that I have to either disable Enhanced Search, or disable Conversations Searchable in the add-on.

 

[Mike]

The search system gets X amount of results and then runs visibility checks on those and displays the ones that you can view. Imagine you do a search where the first 195 results are conversations and the last 5 (of the max 200 results) are posts; there are another 200 references in posts, but they come later in the sorted order (relevance, date, etc). But you aren't in any of those conversations, so they're immediately filtered out. You're now left with the 5 remaining posts (which you can view) so you only see those results. This isn't unique to XF Enhanced Search; this is how XF's search system works. The order differences will likely be the major reason for different behavior.

This can happen without conversations, but conversations just make it worse as 99% of the searchable data isn't viewable once it gets matched.

The only real recommendation I could make is not indexing conversations. You might be able to increase the max search results to mask the issue a bit.

 

[Parsnip]

Thank you for the explanation Mike. I have a lot of member only content so I worry about that affecting things now too.

The more I look into this, the more I see other people having similar issues and I'll take up your suggestion in the appropriate add-on thread.

So in general, if you want to keep conversations indexed and searchable with the benefits Enhanced Search offers, it would seem the workaround is to separate conversations from the default search. It's kind of confusing for users to have them all jumbled up anyway and personally I think that conversations should only be searchable when in a conversation view.

If anyone else has this issue and you don't mind losing the ability to search everything at once, one solution is some template modifications to make the search bar type default to 'post', delete the general advanced search tab, and make the default advanced search instead 'Search Threads and Posts'.

Probably a better fix would be to exclude the 'convess' type, or any non-official types for that matter, from search unless expressly required.

What's the problem with taking permissions into account before the search is done anyway, does it horribly slow things down? It might be nice to at least have a checkbox option for Enhanced Search users if the penalty is not too bad.