Synonym support for XF Search

[Alpha1]

There is a lot of difference between the effectiveness of the XenForo search engine and Google. Google is simply so much better and effective in finding what you need on a XenForo website. IMO it's a difference of night and day.
The main reason is that Google finds synonyms and paraphrasing. The XenForo search function does not.

Please add support for synonyms to XenForo.

Elastic Search allows webmaster to define specific synonyms so the functionality is there. The issue is that it does not have a list of millions of synonym definitions included and there is no way for a webmaster to add this.

I know its probably a complex matter to add this, but IMHO it's a crucial matter.

 

[Alpha1]

Possibly a dictionary API can be used to get millions of synonyms.
http://content.dictionary.com/api
http://developer.wordnik.com/
Or this list from OpenOffice:
http://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries/thes_en_US_v2.zip

 

[Marcus]

I'd pay 50 USD for a Google CSE integration. I took a look into the search handler but it is a bit more complex for me than the other stuff I program for xenForo.

 

[digitalpoint]

Elastic Search already has support for synonyms...

http://www.elasticsearch.org/guide/reference/index-modules/analysis/synonym-tokenfilter/

 

[Alpha1]

Elastic Search already has support for synonyms...

http://www.elasticsearch.org/guide/reference/index-modules/analysis/synonym-tokenfilter/

Yes, indeed. But AFAIK there is no list of synonyms included yet.

 

[digitalpoint]

Yep... You would have to supply a list. It wouldn't make sense to include one globally since what (or how) you want to treat as a synonym is going to vary for each site.

 

[Alpha1]

Such list would contain pretty much all existing words, so its not really possible to compile such list.
Off course there are very specific synonyms for each subject matter, but my suggestion does not pertain to that.
My suggestion pertains to the hundreds of thousands of possible synonyms in common language.


For example: if you search Google for 'newest bulletin board software' then Google will also serve up results with 'latest' 'forum' message board' software.

 

[digitalpoint]

The thing with Google is they don't have a synonym list... rather Google is very good at understanding what the page itself is about. When you base it off a straight word list, you start running into all sorts of problems... you search for "newest bulletin board", and the search parser also thinks you are looking for "contemporary press release plank", and in the end your relevancy of search results is going to go WAY down unless you are *only* calling out the synonyms unique to your site/industry.

It's just not realistic to handle all synonyms of the English language if you ever want to get any sort of relevant search results.

So unless you have a way to understand the context of what a page is about as a whole (like Google does), it would be really bad.

 

[Alpha1]

Good point. Do you think there is a solution to making the search engine understand what a page is about or improving search results in a similar way?

 

[digitalpoint]

Your best bet would be to just use the Google API... But it costs for more than 100 searches a day, and won't search stuff that's not public.

 

[Alpha1]

That doesn't really help then. Hopefully an alternative will come along in time.

 

[Stuart Wright]

Came here to suggest then implementation of synonym support in XF ennhanced search.

Boosting the power of Elasticsearch with synonyms

How to use synonyms and synonym filters in Elasticsearch. Synonyms are a powerful tool for increasing the recall of your search system, but there are many subtleties that are important to know and exp...

www.elastic.co

I would, for example, use it to have TV and television exist as query synonyms.

 

[Alpha1]

@hibiskus please like the first post if you want this implemented. You liked the 2nd post.

 

[Wildcat Media]

Came here to suggest then implementation of synonym support in XF ennhanced search.

Boosting the power of Elasticsearch with synonyms

How to use synonyms and synonym filters in Elasticsearch. Synonyms are a powerful tool for increasing the recall of your search system, but there are many subtleties that are important to know and exp...

www.elastic.co

I would, for example, use it to have TV and television exist as query synonyms.

Since we have a similar forum, there are a lot of terms and acronyms that could be grouped as synonyms, as you can imagine. Members complain about our "awful search" all the time and fully utilizing Elasticsearch's capabilities using synonyms and "fuzzy" phonetic search results I feel are essential to deliver better results to members.