Add-on Sphinx Search Index Add-On

[Wildcat Media]

Since I've gotten almost no feedback for our urgent need to get the Sphinx add-on updated, I am posting here. The current add-on for XF 1.0 is partially functional--it only needs to have the added search options of XF 1.1 integrated into them.

I do not have the luxury of time to code it all myself, although I have started on it. The hardest part is reverse-engineering everything (XF's search, the current add-on, etc.) to get it to work.

 

[agison]

I used to run Sphinx with VB4. It's great search engine with very fast indexing.
But now, Elastic is working fine for me, work with all my & other xenforo addons.

 

[Chris D]

Yeah, I really don't see the need to add another Search system when the built in search is ok (well... better than nothing) and Elastic/XF Enhanced Search is pretty good.

And Dinh Tanh runs one of the busiest XF boards I know so Enhanced Search obviously does well for his site which always has over 10,000 guests online at once.

 

[Wildcat Media]

I do not like Elastic. This has already been debated in other threads.

Anyway, Sphinx is what I am requesting here, so would prefer feedback from those who can help with it. Thanks.

 

[Wildcat Media]

So, nobody can help develop this? I'm really disliking ElasticSearch the more I use it. There is no excuse for anything to hog that much memory (it reeks of sloppy coding), when Sphinx can index 7.5 million posts, return results in under a second, and do so in a much smaller memory footprint. The search results we get with Sphinx are good enough for the general public. What do we do when we hit 10 million posts? 25 million? 50 million? Dozens of GBs of memory to run ES? I don't think so.

May have to find an outside developer to write this. We did buy the ES add-on for XF, only because we are forced into it for having no alternative. It's only temporary. I will be glad to be rid of it.

 

[agison]

So, nobody can help develop this? I'm really disliking ElasticSearch the more I use it. There is no excuse for anything to hog that much memory (it reeks of sloppy coding), when Sphinx can index 7.5 million posts, return results in under a second, and do so in a much smaller memory footprint. The search results we get with Sphinx are good enough for the general public. What do we do when we hit 10 million posts? 25 million? 50 million? Dozens of GBs of memory to run ES? I don't think so.

May have to find an outside developer to write this. We did buy the ES add-on for XF, only because we are forced into it for having no alternative. It's only temporary. I will be glad to be rid of it.

23m posts, 4GB RAM and ES is running fine.
Initial index with ES is so long (a half of day), but since that time, didn't care about it anymore.

 

[Brent W]

So, nobody can help develop this? I'm really disliking ElasticSearch the more I use it. There is no excuse for anything to hog that much memory (it reeks of sloppy coding), when Sphinx can index 7.5 million posts, return results in under a second, and do so in a much smaller memory footprint. The search results we get with Sphinx are good enough for the general public. What do we do when we hit 10 million posts? 25 million? 50 million? Dozens of GBs of memory to run ES? I don't think so.

May have to find an outside developer to write this. We did buy the ES add-on for XF, only because we are forced into it for having no alternative. It's only temporary. I will be glad to be rid of it.

I would prefer this as well, only because I also use vBulletin 3.8, 4.x, IPB and xenForo. xenForo is the only one that doesn't support Sphinx

 

[Wildcat Media]

We are actually going to make the move (finally) to XF from vB 3.7 within the next 48 hours.

I can understand the reasons why ES was chosen over Sphinx, but it is impractical, especially in terms of memory usage. That and it stalls--I have to restart it manually a few times per day (and this is just on our test forum where it's hardly used at all). Sphinx has run for several months with no interference. It's been out longer, it's proven, and it can handle far larger data sets on fewer resources. If Craigslist uses it without issue, then I think it would still do just fine for a little forum like ours.

And with the new live updating, there is no longer a need to rebuild indexes nightly. That is why I think someone could reverse engineer the ES add-on for XF and rewrite it to use Sphinx. There is a Sphinx add-on, but in its current state, it only works on having Sphinx generate delta indexes every couple of minutes, and doing a nightly rebuild (which is needed to flush out indexes that point to deleted records in the tables).

I just wish I knew enough to write it myself. I'd sell it as an add-on. I'm good with writing my own PHP, but having to learn how XF is coded is just outside my sphere of knowledge right now.

 

[HWS]

While I also would prefer Sphinx over ES, if it would be available, there are other things in a XenForo installation at a big board that warrant more attention.

 

[Wildcat Media]

While I also would prefer Sphinx over ES, if it would be available, there are other things in a XenForo installation at a big board that warrant more attention.

I agree there are likely a few issues that could address big boards like ours, but I also can't fault XenForo since it is a relatively new product and I can't expect it to be a Swiss Army Knife of configuration options at this point. But we do what we have to, and we get by. My staff is already accomodating the changes we need to make so that XF works for us.

Just the basic architecture of XF alone is a lot more big-board-friendly than vBulletin ever could be. I even recall vB's attitude was such that (especially after IB took over) they felt big boards were such a small percentage of customers that they had no motivation to do anything about it. Yet I will also say that a few vB staffers did take time to help us out with our load issues, especially with server configuration, and we learned a lot from it all. The long-running "big board" thread was also a treasure trove of information and is what led me to Sphinx in the first place, along with learning about opcode caches and what not.

Speaking of big boards, I'd love to see a separate forum area here at XF, or at least a thread prefix, dedicated only to big board operators. Even with search running well here, I still can't afford reading through search results for an hour trying to find a snippet of a conversation I read several months ago. I would not even mind setting up a site with some basic XF big board tweaks just so we could use it for reference.

And while I'm still wishing...yes, I'd like a proper Sphinx add-on.

 

[Wildcat Media]

Well, my dislike of ElasticSearch grows with each passing day. It only runs two or three days before stopping dead. Its logs are useless, hundreds of thousands of lines long basically telling me nothing as to why it aborted. We've already bumped up the open files limit beyond what I feel safe doing. I have double the memory set aside for what it uses. I've done everything else recommended that I could find on the Web (which is not much--their own site is woefully inadequate in documentation). And yet it still crashes, and always needs to be started manually each time or it will not stay running. Even with monit, it will restart and die within 60 seconds.

I'm either going to have to use the ES add-on to develop my own Sphinx add-on, or contract outside the forum here to get it done. It's now at the "I'm sick and tired of messing with this crap" stage. I used to run Sphinx for years, and had it stop running maybe only once or twice per year. I thought ES was half-baked when I saw that it ran on top of Java; after having used it, I can see why it's not very popular or well-documented out there on the Web: it's just not good. Fixing ES is not an issue anymore: I'm done with it.

So I either try to rejuvenate the old Sphinx and run delta indexes once every minute or two, or jump head-first into Sphinx's new SphinxQL, rewrite it using the ES add-on for examples on how to interface an external search index with XF, and have it update the search indexes with each post. I actually prefer having it index once per minute or two--if it ever goes down, simply running a delta index brings it up to date. When ES fails, I have to restart it manually, then wait until off-hours to once again rebuild the search indexes.

Had I known, I would have gone with another forum system that supported Sphinx (other than vB of course).

 

[Chris D]

Have you raised a support ticket?

Your feedback doesn't echo that of others I've read. Perhaps there is actually a problem that can be solved, and considering you paid money for the license, you might as well use the support you're entitled to.

 

[Brent W]

I wonder if DigitalPoint is going to use ElasticSearch or create a SphinX solution the way he did for vBulletin 4?

 

[Wildcat Media]

Have you raised a support ticket?

Your feedback doesn't echo that of others I've read. Perhaps there is actually a problem that can be solved, and considering you paid money for the license, you might as well use the support you're entitled to.

The problem is with ES, not XF--they shouldn't have to support crappy coding by someone else. I even started over and used some of Jaxel's installation instructions here (we already have Java installed), yet the results were the same. ES is just an immature product. I figure if Sphinx can handle Craigslist, it can handle our dinky little 8 million post forum.

No, I'm done with ES. I've wasted countless hours messing with it, so it's time to move on. I do know that if I have to work on the add-on myself, it'll be offered up as a paid add-on here if I feel I can support it. It'll have to wait a few weeks until I can find time to figure out where to start on this project. Too much going on right now.

 

[Wildcat Media]

I wonder if DigitalPoint is going to use ElasticSearch or create a SphinX solution the way he did for vBulletin 4?

I'm all ears if he goes with Sphinx...it shouldn't be that hard to get working. The existing add-on actually worked great, except that with XF 1.1, there are more areas that can be searched and the old add-on does not support them (profile posts, etc.). The parts it did work with, it worked great. And I noticed that ES lags, where Sphinx was more instantaneous. Kind of expected when you have to run something on Java.

BTW, I'd love to add the ability to search private conversations...

 

[Slavik]

I'm all ears if he goes with Sphinx...it shouldn't be that hard to get working. The existing add-on actually worked great, except that with XF 1.1, there are more areas that can be searched and the old add-on does not support them (profile posts, etc.). The parts it did work with, it worked great. And I noticed that ES lags, where Sphinx was more instantaneous. Kind of expected when you have to run something on Java.

BTW, I'd love to add the ability to search private conversations...

Shawn has already commented elsewhere that if he were to build his own system it would most likely be build on ElasticSearch also.

As for your problems, I can only summise its a server configuration error with your setup. I have been running ES without fault and with only 1 restart (which was when I changed some settings) for coming up on a year now on P8ntballer, otherwise the service has been running none-stop.

I also have installed ES on several servers for people, and they all work perfectly fine.

As for your Sphinx solution, all the Enhanced Search addon does is act as a middleman between the database, the search function and the ES install, using that you should be able to modify it relatively easily to pass and receive the data from Sphinx instead.