Confirmed Stop Word Problem

rfc0001

Active member
#1
Hi, I have several common words added as "stop words", which Enhanced search describes as words not included in the index.

I assumed these words were also ignored in searches. E.g. If I search for "in the world" and in and the are stop words, Xenforo would just search for "world".

However, I've found out that in the above example "in the world" returns 0 results, "the world" returns 0 results, and "world" returns several results.

What I assume is happening is XF ES is actually searching all three terms including stop words, and since 2 of them are stop words (therefore not indexed) XF is getting 0 results for those. I think this behavior is a bug. I would expect XF to exclude stop words from both the index and when performing searches. As it is, all stop words do is literally stop the search and return 0 results, which I don't think is very useful or the expected behavior.
 

Mike

XenForo developer
Staff member
#2
After you updated the stop words, did you rebuild the index? If you don't mind, can you try restarting Elasticsearch?

Stop words should be handled entirely internally within Elasticsearch, so it handles ignoring them when doing analysis, both for searches and indexing.

Saying that, are you doing a phrase search (with actual quotes)? If so, this may be how Elasticsearch actually works. It has to do with how they do positional data and the stop words reducing precision/information.
 

rfc0001

Active member
#3
I did rebuild the index, deleting it first. I'll try restarting elastic search. I just uninstalled/reinstalled it Wednesday. I'm not doing a phrase search (no quotes).
 

rfc0001

Active member
#4
After you updated the stop words, did you rebuild the index? If you don't mind, can you try restarting Elasticsearch?

Stop words should be handled entirely internally within Elasticsearch, so it handles ignoring them when doing analysis, both for searches and indexing.

Saying that, are you doing a phrase search (with actual quotes)? If so, this may be how Elasticsearch actually works. It has to do with how they do positional data and the stop words reducing precision/information.
I completely uninstalled/reinstalled Elasticsearch and Enhanced Search, rebooted the server, and rebuilt the index and still having this issue. Searches with stop words return 0 results. If I omit the stop words, the search return results. If I disable stop words, the original search returns results.
 

Mike

XenForo developer
Staff member
#5
I've managed to confirm this now. However, testing across multiple versions, it appears to only happen in Elasticsearch 6. I'll move this to bugs, but I'll need to do more testing to see if this is something on our end or an Elasticsearch bug. There are significant changes both within XFES for Elasticsearch 6 (and thus likely large changes within Elasticsearch itself), so it certainly could be anywhere.

As it stands, the only workaround I could recommend is not using stop words for now.
 

rfc0001

Active member
#6
I've managed to confirm this now. However, testing across multiple versions, it appears to only happen in Elasticsearch 6. I'll move this to bugs, but I'll need to do more testing to see if this is something on our end or an Elasticsearch bug. There are significant changes both within XFES for Elasticsearch 6 (and thus likely large changes within Elasticsearch itself), so it certainly could be anywhere.

As it stands, the only workaround I could recommend is not using stop words for now.
Thanks. My workaround is to disable stop words, which it appears XF Community has done as well.
(Assuming you are using Enhanced Search) 👍.
 
Top