@Andrej my add-on ElasticSearch Essentials does a bunch of acrobatics to mostly avoided this is by pushing user contextual information into the general search.
It requires Elastic Search and opt-in per content type but I've implemented the core XF contene types + conversation messages (if you can see them) from my own add-on Conversation Improvements.
The major issue is the XenForo search query language really isn't expressive enough to handle this well, and most fo the content types do not push enough information into the search system to exclude things the searcher has no hope of seeing.