ES 2.2 Enhanced Search 2.2 - Similar Threads

Screenshot 2020-07-20 at 13.37.50.png
In the beginning, XenForo Enhanced Search (XFES) was intended to be a simple drop-in replacement for MySQL full text search, primarily aimed at large forums whose operation would potentially suffer when attempting to search very large database tables.

As a drop-in, XFES only made use of the enhanced search index and speed for the formal search system, but with version 2.2 we are starting to leverage more of the power afforded by the Elasticsearch back-end.

Similar threads​

A useful tool for surfacing content on your forum is a list of Similar or related threads, which can be displayed along with a primary thread. Visitors who have landed on that page from a search engine will often look for related material to continue their reading, and Similar threads offer a great way to provide that in an automated fashion.

One limitation of the usefulness of Similar threads is a determination of just how similar or related the results actually are. This is where XFES comes into its own, as the Elasticsearch back-end provides a specific ability to search for more like this using the current thread as a search parameter. Given a decent amount of content from which to pluck similar results, this provides an excellent list of threads in which the viewer of the current thread may be interested.

Similar threads widget​

We display our similar threads results in a widget, and by default we place it below the quick reply area on thread view pages, but as it is a widget you can choose to place it wherever you like on the page.

Screenshot 2020-07-20 at 13.37.50.png
Screenshot 2020-07-20 at 13.33.31.png


You could even choose to place multiple instances of the widget on the page, and make use of the power of the widget system to control how each instance of the widget behaves, setting parameters such as the number of thread results visible, the style of display, the source forums searched and the time period within which to search.

Screenshot 2020-07-20 at 13.52.22.png
Screenshot 2020-07-20 at 13.44.36.png

Controlling the index​

Of course, it's possible to fine-tune the way in which results are found, and we provide options to positively-weight results based on being in the same forum as the source / original thread, and based on how new the related content may be.

Screenshot 2020-07-20 at 13.54.46.png

But wait, there's more...

The extreme speed and efficiency of XFES allows us to offer another, related tool to our arsenal: similar thread suggestion at thread composition time.

Suggest while composing​

Everyone hates being told that a question they've asked has already been dealt with, or that the topic about which they have posted has already been covered.

XFES 2.2 aims to minimise the occurrence of this, by suggesting related threads that may contain content similar to that which the user is about to post, while they are composing. XFES is fast enough that sensible suggestions can be made in real-time while the user is typing the title of their new thread, which may prompt them to visit one of the suggested existing threads rather than posting a potential duplicate.

Screenshot 2020-07-22 at 11.05.41.png Screenshot 2020-07-22 at 11.38.43.png Screenshot 2020-07-22 at 11.40.09.png​
The thread suggestion system can be a real game-changer for unwanted duplicate threads.

The system can be turned on and off through the control panel, along with options for how many similar threads to suggest.

Screenshot 2020-07-20 at 14.40.13.png

It's also possible to prevent the suggestion system from appearing at all on a per-forum basis. For example, we would never want to be pestered with suggestions for similar threads when composing a new release announcement, or a Have You Seen thread.

Screenshot 2020-07-20 at 14.42.39.png

In conclusion​

We think you'll agree that the addition of Similar threads functionality to XenForo Enhanced Search adds a compelling new ability to an already very powerful system.

By way of a reminder, XFES requires that your server has Elasticsearch 2.0 or newer, and can be purchased here.
 
Click to expand...
ALMUSA said:
Is there a field where we can omit certain phrases/words not to be included when showing results of similar threads e.g and, or, when, what, on, it,?
Click to expand...
Mr Lucky said:
Will this include the ability to list very common stop words? For example in a forum based around Logic software, we would want the word logic to be excluded?
Click to expand...
This is crucial, not just for general common words, but also specialist common words within your niche.

You get a lot more relevant hits once the common words are excluded via a stop word list - which you can gradually refine over time (this is what I have learned from using the similar threads add on)
 
Kirby said:
Does that answer your question?
Click to expand...
Thanks, I'd forgotten about the stock ES stop words. Indeed it partially answers it in that is for the index in general, I was thinking more purely in regard to the similar threads suggestions.

I think that is fine for the general common words, but (from my experience with Similar Threads addon) certain niche words I've found are useful to exclude from a widget only showing a few results, as opposed to general search showing many results.
 
Last edited:
BIG LLC said:
Can we limit the date range? So a thread from two years ago would probably not be relevant, but one from two days ago almost certainly would.
Click to expand...
There isn't an explicit date option, but it follows the recency-weighting option which applies to general searches as well, so that more recent results have a boost in relevancy which will tend to suggest them unless there's a noticably better older result.
 
Mr Lucky said:
This is crucial, not just for general common words, but also specialist common words within your niche.

You get a lot more relevant hits once the common words are excluded via a stop word list - which you can gradually refine over time (this is what I have learned from using the similar threads add on)
Click to expand...
Realistically, there will likely be tweaks and adjustments to the similar threads "algorithm" through the beta process. This may mean explicit similar threads stopwords being supported or some other options as well. (I put "algorithm" in quotes there because what we'd be tweaking is more likely to be the configuration to the algorithm than the algorithm itself.)

Saying that, I don't think stopwords are necessarily as significant as you might think due to how the relevancy system works. As always, the Elasticsearch docs are pretty inscrutable, but there's some discussion of the approach here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html The rough summary of the relevant part is that because of the TF-IDF algorithm, more common words will generally affect relevancy less. And depending on the size of the input document, they may not even be part of the similarity match.
 
Mike said:
Saying that, I don't think stopwords are necessarily as significant as you might think due to how the relevancy system works.
Click to expand...
Fair enough, but I'm just thinking of the case where I would want certain words to be included in the general search, but not the similar threads suggestions. It would be a great algorhithm that can read my mind. Or not.

And I may well be barking up the wrong tree basing this on my experience with the the Similar Threads addon. (And it will be interesting to do a comparison)

Plus, I can see the advantage in keeping it simple!
 
Last edited:
Some great additions!

But where is the search bar auto complete. Missed opportunity to not include that, so still having to rely on 3rd party addons just for that important function. :\
 
I would love to see a cache option with the similar threads. Just update the similar threads once in 30 days for example so the pages will be loading fast.
 
14 days cache is already there. One of the devs confirmed in another thread. Let me see if I find the link!

xenforo.com

Implemented - Enhanced Search - Similar Thread Caching

For larger forums with millions of threads, it would be very helpful to have similar thread caching for results (i.e. cache similar threads for 30 days) - ideally using a field in the xf_thread database, so that it works for long term caching. There are millions of threads which are old and...
xenforo.com xenforo.com
 
Where do we enable the similar threads? I see nothing in the Enhanced Searh Options, and find nothing if I search ACP for similar.

I had this working on a previous beta, but not beta 3.
 

Similar threads

X
XenForo search errors instead of reporting the search is too long
Replies
0
Views
204
Xon
X
K
XFES ignores setting searchMinWordLength
Replies
1
Views
189
Xon
X
K
XFES doesn't warn user about stop words exclusion
Replies
1
Views
141
Xon
X
Ludachris
  • Suggestion Suggestion
Secondary Similar Threads function - manually curated
Replies
0
Views
341
Ludachris
Ludachris
Recep Baltaş
  • Solved
XF 2.2 Elasticsearch error: no such index
Replies
12
Views
1K
noxa
N
Top Bottom