1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ES vs. Sphinx

Discussion in 'Enhanced Search Support' started by Gladius, Feb 10, 2012.

  1. Gladius

    Gladius Well-Known Member

    Any direct comparisons so far? From what I've read, Sphinx is much more efficient in several (all?) aspects than ES so it makes me wonder why XF's gone with ES...
    TheVisitors likes this.
  2. Slavik

    Slavik XenForo Moderator Staff Member

    New informationssssssssss
    Dinh Thanh likes this.
  3. Gladius

    Gladius Well-Known Member

    I don't know, most of us are rather sensitive about how much CPU and RAM resources are wasted and ES uses up a lot more of both from the results I've seen posted here (considerably more HD space as well). For our uses, the ES benefits are secondary to Sphinx's so I'd certainly prefer Sphinx integration over ES.
    TheVisitors likes this.
  4. Slavik

    Slavik XenForo Moderator Staff Member

    You will most likely want to have a private developer update Sphinx for you then.

    Xenforo chose Elasticsearch over Sphinx for good reason :)
  5. Gladius

    Gladius Well-Known Member

    Because it was easier? Honest question, because I just don't see how ES would beat Sphinx for the majority of big board users.
    TheVisitors likes this.
  6. Slavik

    Slavik XenForo Moderator Staff Member

    If you cant see the benefits then you need to do more research ;)

    To list off just a handful.

    ES has out the box support for clusers (which big boards are more likely to use). This is possible in Sphinx, however it is considerably more complex to set up.

    Sphinx indexs documents with a large delay (deltas). ES has a default delay of one second.

    ES has built in "more like this" searches (document comparison). Sphinx doesn't.

    ES is easier to set up and get running for end users.

    ES is easier to integrate with as it uses JSON.
    D.O.A. likes this.
  7. Gladius

    Gladius Well-Known Member

    I'm not sure how doing more research would change the facts that I've stated in this thread (as reported by admins who've tried both Sphinx and ES), unless you dispute their validity? I get it that ES comes with more extras that may or may not be useful to some, but the core aspect that big board admins look at is performance in conjunction with resource utilization. Everything else is secondary to that under normal circumstances.
    TheVisitors likes this.
  8. Slavik

    Slavik XenForo Moderator Staff Member

    Because all your looking at is resource usage and not functionality.

    But as said, I would suggest if you want to use sphinx, you look at hiring a developer to update it for you.

    Obviously only Kier or Mike can decide and comment on this, however I would be confident in saying XenForo won't be releasing a Sphinx search solution anytime soon. Their current search solution for big boards works, and works fantastically, their resouces will be better spent developping the core product.
    D.O.A., Dinh Thanh and ankurs like this.
  9. digitalpoint

    digitalpoint Well-Known Member

    Either one would have been fine really... As far as the argument that Sphinx results are delayed, that's only the case if you set it up that way.

    Realtime indexes work just fine and is even simpler than using a JSON REST request to do it: http://sphinxsearch.com/docs/current.html#rt-indexes

    You can simply use the normal MySQL protocol to do a DELETE, INSERT or UPDATE to the Sphinx index.

    I'm not opposed to Elastic Search, just wanted to clarify some incorrect information being spread.
    Rudy, Adam Howard and Gladius like this.
  10. Gladius

    Gladius Well-Known Member

    Thanks digitalpoint, I always appreciate your expert input. For your site, would you pick ES over Sphinx?
    TheVisitors likes this.
  11. digitalpoint

    digitalpoint Well-Known Member

    If I had to build it from scratch as of today, I probably would do it with Sphinx. Not because I think Sphinx is necessarily better, but only because I have more experience with it, so my development learning curve would be shorter.

    As far as which I consider to be fundamentally better... I can't really give an educated opinion since I just don't have enough experience with Elastic Search (as a developer or as a user) to make an actual determination.

    That being said, for XenForo, I plan on using Elastic Search since I don't have to build it or maintain it. And once I have a live site using it in the real world (and I've done whatever tuning I need to do), I'll be able to have a better opinion on it vs. Sphinx.

    Bottom line is something based on Elastic Search OR Sphinx is going to be infinitely better than a FULLTEXT search engine based on MySQL.
    simbolo, el canadiano, D.O.A. and 2 others like this.
  12. Gladius

    Gladius Well-Known Member

    Ah, well, at least you using it in the future is some consolation that whatever can be optimized with ES, you'll do it and hopefully share with us. :D
    TheVisitors likes this.
  13. Adam Howard

    Adam Howard Well-Known Member

    Playing with a developer snap shot of Sphinx that has a prototype of real time indexing. Have no idea of this will be released to the public, but I can't imagine it wouldn't be.
  14. digitalpoint

    digitalpoint Well-Known Member

    Sphinx realtime indexes have been in the latest non-dev version for awhile:


    Realtime indexes still have some limitation though... maybe you got a dev version that fixed those limitations? Most notably (for me anyway) realtime indexes don't support MVAs or prefix/infixs yet.
  15. digitalpoint

    digitalpoint Well-Known Member

    Okay... I have a *little* more experience with ES now... And the more I play with it, the more I like it. It does some things that I wish Sphinx would do... for example it's ability to auto-shard and replicate to other nodes (servers) is pretty seamless and awesome (it's actually the easiest system of any sort I've dealt with for sharding/replicating... it more or less "magically" works just because it's on the same local network as other nodes... add a server, and the other nodes instantly distribute data to the new node as soon as it's online).

    I didn't like how it was treating punctuation as part of a word by default, but that was easily enough fixed: http://xenforo.com/community/resources/change-analyzer-for-enhanced-search.643/

    I also think indexing 1 and 2 letter words like it does by default is a rather pointless waste of disk space/memory... but that was also easily fixed in the config file.

    I haven't run benchmarks or loaded a ton of data into it yet... and also don't know how the memory requirements compare to Sphinx now that I stopped indexing words < 3 characters.

    I ended up writing a module so I can monitor the ElasticSearch cluster from the XenForo admin home...


    If the benchmarks end up being as good as the rest of it, I'll be sold on ES over Sphinx.
    Jarod, Walter, Gladius and 2 others like this.
  16. Dinh Thanh

    Dinh Thanh Well-Known Member

    I love your ElasticSearch Statistics. Very nice.
  17. digitalpoint

    digitalpoint Well-Known Member

    Not really much to look at currently... just a dev setup without much content. :)
  18. p4guru

    p4guru Well-Known Member

    awesome stuff Shawn

    looking forward to more comparisons between sphinx and ES :)
  19. digitalpoint

    digitalpoint Well-Known Member

    So I ended up making that little status block for ElasticSearch so you can click the nodes to get detailed info about that node... Might help finding config issues fairly easy...

  20. Slavik

    Slavik XenForo Moderator Staff Member

    Would love to have these :eek: Any chance of releasing them ?
    Walter likes this.

Share This Page