How to get the best Similar / Related Threads? A discussion about the search.

Stuart Wright

Well-known member
I posted in my suggestion for a better sidebar system here that I think it's important that we have a display of similar threads in every thread displayed.
Big sites often have a huge bounce rate. Most people land on your forum via a search engine directly into a thread, do or don’t find the specific information they want and then click the Back button. If there was a block showing related content (similar threads, showcases, media gallery items etc.) then the visitor is more likely to click one of those and stay at your site. This is potentially a very useful addition for retaining users, increasing impressions and reducing the bounce rate.
I probably understated how important I think this is. If, like us, you have a bounce rate of around 70% then you *have got* to do something to keep people at your site once they have found you. Every time you don't, there is a lost opportunity to gain a new member.

So the question here is how to return the most relevant set of search results based on the thread title.

The only addon I have found which does this is @Daniel Hood 's addon for [bd] Widgets, [XenMods] Similar Threads. I tested this but the search results were too generic and not similar to threads I was viewing. Daniel is looking in to how to make the similar threads search better and I wanted to put the question out to the development community here at Xenforo. Not because I don't think Daniel can do a great job, but because this functionality is so important that it should be in the core software and I think that the idea will benefit from as much input as possible.

There is a Similar Threads addon by @AndyB which displays them when creating a new thread. The idea being to avoid posting threads similar to existing ones. Andy has included some options which help narrow down the search results. He is searching on the thread title only, of course.
  1. Limit search to threads in the same forum
  2. Exclude specific forums from the search
  3. Minimum word length - ignore words shorter than a specified number of characters
  4. A list of common words to exclude from the search
  5. Specific punctuation characters to ignore
  6. Support for multibyte characters - I guess this is an option in the search, but I don't know anything about it
There are good ideas here.

I'd like to run through my thoughts and then invite your feedback.
First how would the search be influenced by the presence of Xenforo's Enhanced Search addon? I'm guessing a fair bit of added functionality is available in Elastic, but I'm unfamiliar with what this is so I'm going to speak generally and assume that the search could be changed to take advantage of Elastic if it is installed.

Keyword matching
I have a problem with excluding words based on the number of characters. If, for example, we have a title of 'What is the best LG TV?' the two letter words LG and TV are the most important search terms for finding a similar thread.
But obviously it's important to give less weight to common words. This document http://www.elasticsearch.org/guide/...nce/current/query-dsl-common-terms-query.html implies that with Elastic, we can nail this. If Elastic figures out the common terms itself based on what's in the index, and then allows us to give them less weighting, then that's awesome. But if the Xenforo installation does not include Elastic, then would there need to be a list of common words entered, I guess, in order to try and achieve a similar result.

Forum
I think the weighting of the search results should be influenced by the forums they are in. So threads in the same forum should be most given preference. Then the child forum, sibling forum and parent forum. In this order? In any order?
Once the above is in place, I can't imagine a scenario where I would want to exclude the results from a specific forum. If I was looking for a similar thread to one in the OLED TVs forum, I can't imagine I would want any results from the Holidays forum, but if preference is given to threads based on the forum then hopefully that won't happen.

Characters
With regard to punctuation characters, the only one I can think would make any difference is the question mark. I'm thinking that if the thread is a question, then we'd want to match similar questions.

Thread date
Should any preference be given to more recent threads?

Prefix
If the thread has a prefix, then some preference should be given to threads with the same prefix.

Is there anything else which would help find the best set of search results?
Any input from you?
Thanks
 
thread tags are an excellent indicator for similar threads.
We don't have thread tags on AVForums. Even if we had that addon, we couldn't rely on people using it in order to return the best search results. If every thread was tagged appropriately then great. But unfortunately not.
 
  1. Limit search to threads in the same forum
  2. Exclude specific forums from the search
  3. Minimum word length - ignore words shorter than a specified number of characters
  4. A list of common words to exclude from the search
  5. Specific punctuation characters to ignore
  6. Support for multibyte characters - I guess this is an option in the search, but I don't know anything about it

Hi Stuart,

The Similar Threads add-on v3.6 is located here:

https://xenforo.com/community/resources/similar-threads.2441/

The default XenForo search or as I call it the Standard Search will work with Similar Threads, but the new version of Similar Threads now also supports XenForo Enhanced Search. I quoted the above to bring to your attention that options 3-6 are applicable only to the Standard Search. Similar Threads using Standard Search is somewhat limited, but it's designed to be used on newer forums and for that it works well enough. On forums with more than 250,000 posts, I suggest using XenForo Enhanced Search, this will make Similar Threads return much better results.

Regarding the Same Forum feature, I think on larger forums this is a must. Forums should be well organized so that Similar Threads will provide great results.
 
Maybe a stupid one but whether the person searching has replied to another thread should maybe add (or subtract) weight to it? Likewise if someone they follow has posted in it?
 
I think that the search results should focus on the content of the thread title because it's the thread title which will have landed people in the thread on your forum, and so it's this which they will be interested in. If the contents of the thread they are viewing does not match their expectations, unless there is an enticing link to a similar titled thread obviously visible, they will just click Back.
The 'popular thread' weighting idea I like, but I think the search results should not relate to the person viewing. Most of the time it will be a guest, anyway.
 
Last edited:
The 'popular thread' weighting idea I like, but I think the search results should not relate to the person viewing. Most of the time it will be a guest, anyway.
You could related thread popularity not to the person viewing, but to those who have already replied to the viewed thread. In other words, you could give other threads a higher weight where one or more participants of the viewed thread have also participated in, assuming that in a big board, members usually hang out in "their" related areas. It's not conclusive, but it's an extra attribute you could use.
 
You could related thread popularity not to the person viewing, but to those who have already replied to the viewed thread. In other words, you could give other threads a higher weight where one or more participants of the viewed thread have also participated in, assuming that in a big board, members usually hang out in "their" related areas. It's not conclusive, but it's an extra attribute you could use.
Yeah that's an interesting idea.
 
I think that the search results should focus on the content of the thread title because it's the thread title which will have landed people in the thread on your forum, and so it's this which they will be interested in. If the contents of the thread they are viewing does not match their expectations, unless there is an enticing link to a similar titles thread obviously visible, they will just click Back.
The 'popular thread' weighting idea I like, but I think the search results should not relate to the person viewing. Most of the time it will be a guest, anyway.
I agree with this, especially after getting the add on. Large threads show up because the key words are in there somewhere, but not as relevant often as a shorter thread but with the key words in title. Plus unless the thread looks very relevant, lots of people aren't going to sift through hundreds or thousands of posts.
 
Last edited:
The only addon I have found which does this is @Daniel Hood 's addon for [bd] Widgets, [XenMods] Similar Threads. I tested this but the search results were too generic and not similar to threads I was viewing. Daniel is looking in to how to make the similar threads search better and I wanted to put the question out to the development community here at Xenforo. Not because I don't think Daniel can do a great job,

BTW, I have bought Daniel Hood's addon and I think it is very good and can only get better, hence I thought it was worth paying for to encourage development and highly recommend it. I especially think the widget is useful. We just need more search options,
e.g.
  • Exclude specific words
  • Exclude common words
  • Search title only on/off
  • Extra Weight to newer threads on/off
  • Enable search current forum on/off
Thread date
Should any preference be given to more recent threads?

Only if it is an option. I like that sometimes very old threads are found and given a new lease of life.
 
Last edited:
Back
Top Bottom