Digital Doctor
Well-known member
It would be nice to know what it is
http://xenforo.com/community/threads/development-update-may-2011.16084/#post-211755
Sphinx Search Engine
- Thread starter mlx
- Start date
http://xenforo.com/community/threads/development-update-may-2011.16084/#post-211755
Wildcat Media
Well-known member
I wonder how easy it would be to switch from the current Sphinx add-on to this new one. I am considering XF for the big board, and would be happy if either one worked...but do prefer an "official" add-on via XF of course. Sphinx was a huge improvement for vB3.7.
lazy llama
Well-known member
Would there be any chance of this being updated for v1.1?
At the moment, search still works on 1.1 beta 1 but it doesn't recognise the "Search this thread only" limit.
At the moment, search still works on 1.1 beta 1 but it doesn't recognise the "Search this thread only" limit.
lazy llama
Well-known member
Excellent. If you need a stupidly large test site, let me know.
mlx
Well-known member
Really sorry I didn't have time to update the add-on yet. As we are planning to finally migrate our vBulletins to XF v1.1 there will be an update eventually. Unfortunately this is pretty much our hobby and I didn't have much free time lately. So just in case anyone wants to help developing this or wants to branch it out, feel free to use my code any way you want!
Wildcat Media
Well-known member
If I had the time and the brains to do it, I would take over. (I am a decent PHP coder but don't have dozens of hours to figure out how XF works behind the scenes.) I've seen what Sphinx did to our vB installation, and it was jaw-dropping.
Since it seems the big board search isn't forthcoming (unless I missed it--I'd seen the question asked but never saw an "official" reply) and this add-on won't work in 1.1 yet, we're looking at another upgrade path for our "big board" now. No enhanced big board search is the major deal killer for us, and I want vB's code off of our servers as soon as I can get something better in its place. I don't want anything from that despicable company on our server anymore...
Agusti Fernandez Pardo
Member
@mlx, I sent you the email about the possibility of helping you monetary speaking on the behalf of this mod.
Rudy, I would really appreciate a way to get in touch with you, could you please start a conversation with me? Thank you
Rudy, I would really appreciate a way to get in touch with you, could you please start a conversation with me? Thank you
Wildcat Media
Well-known member
We're probably going to move on and go with another system. We can't afford to wait any longer for "maybes." Thanks anyway.
Jake Bunce
Well-known member
Got a request to open this thread for discussion about revamping this addon, so I have moved it to discussions.
Ok discuss. Go!
Ok discuss. Go!
Wildcat Media
Well-known member
Thanks much Jake! Discussion is a "go." 
I do not have it working on a test instance of 1.1.2 right now (I was hoping to duplicate lazy llama's partial success with 1.1b1). I have the latest Sphinx installed (2.0.4-release), and it is indexing the forum posts and threads with no issues. I can search via the command line and get the right results, but something is not making it back to XenForo. I did see an earlier post in this thread (must be on the prior page) about XF working fine with Sphinx 2.0.1-beta; 2.0.4 being only a minor point release should not introduce anything major that would impede us getting search results.
I have talked privately with mlx, and we are free to build onto the existing add-on. From what I can tell, the Sphinx.php file is the one we would need to touch up. This is the one that generates the actual queries sent to the Sphinx search engine. The Sphinxapi.php file looks like the bridge between Sphinx and XenForo (can't think of another way to explain it, as it obviously addresses the Sphinx API).
I don't know which content types are being referred to here, but it is rather easy to index any additional tables in Sphinx by adding onto the sphinx.conf file.
My question here is this: what other content types should Sphinx index, or does it need to? Would XF's built-in search still handle those if Sphinx doesn't, or is this Sphinx search add-on completely replacing all searching in XF?
The reason I'm posting here now is this: I'm sure there are enough of us here that could get this working, as I feel there is likely not much work that needs to be done. I am betting the Sphinxapi.php script is pretty much set in stone, and we would only need to add the desired additional search parameters to Sphinx.php. (At least, that is my rough understanding after a cursory look at these two files.) Ideally the add-on would be rewritten to use the instant update features of Sphinx, but that is beyond the scope of this add-on for now. If I had the coding skills to do this, I wouldn't even be asking here.
Finally...yes, I know about ElasticSearch, but its major shortcoming is memory usage. We have used Sphinx for three years, now indexing 7.5 million posts, and it still returns results lightning fast (0.76 seconds for my last search) being accessed from disk. No need for a lot of memory IOW. If anyone wants to debate ES vs. Sphinx, feel free...but please, don't threadcrap: start another thread or choose one of the existing threads. I only want to see if it is feasible to revamp this add-on for the time being, even if we have to put in a few bucks toward development of it down the road. I feel this add-on is just missing a few components that would make it work in 1.1 just as well as it did in 1.0.
I do not have it working on a test instance of 1.1.2 right now (I was hoping to duplicate lazy llama's partial success with 1.1b1). I have the latest Sphinx installed (2.0.4-release), and it is indexing the forum posts and threads with no issues. I can search via the command line and get the right results, but something is not making it back to XenForo. I did see an earlier post in this thread (must be on the prior page) about XF working fine with Sphinx 2.0.1-beta; 2.0.4 being only a minor point release should not introduce anything major that would impede us getting search results.
I have talked privately with mlx, and we are free to build onto the existing add-on. From what I can tell, the Sphinx.php file is the one we would need to touch up. This is the one that generates the actual queries sent to the Sphinx search engine. The Sphinxapi.php file looks like the bridge between Sphinx and XenForo (can't think of another way to explain it, as it obviously addresses the Sphinx API).
I don't know which content types are being referred to here, but it is rather easy to index any additional tables in Sphinx by adding onto the sphinx.conf file.
My question here is this: what other content types should Sphinx index, or does it need to? Would XF's built-in search still handle those if Sphinx doesn't, or is this Sphinx search add-on completely replacing all searching in XF?
The reason I'm posting here now is this: I'm sure there are enough of us here that could get this working, as I feel there is likely not much work that needs to be done. I am betting the Sphinxapi.php script is pretty much set in stone, and we would only need to add the desired additional search parameters to Sphinx.php. (At least, that is my rough understanding after a cursory look at these two files.) Ideally the add-on would be rewritten to use the instant update features of Sphinx, but that is beyond the scope of this add-on for now. If I had the coding skills to do this, I wouldn't even be asking here.
Finally...yes, I know about ElasticSearch, but its major shortcoming is memory usage. We have used Sphinx for three years, now indexing 7.5 million posts, and it still returns results lightning fast (0.76 seconds for my last search) being accessed from disk. No need for a lot of memory IOW. If anyone wants to debate ES vs. Sphinx, feel free...but please, don't threadcrap: start another thread or choose one of the existing threads. I only want to see if it is feasible to revamp this add-on for the time being, even if we have to put in a few bucks toward development of it down the road. I feel this add-on is just missing a few components that would make it work in 1.1 just as well as it did in 1.0.
Wildcat Media
Well-known member
OK, I solved the problem of Sphinx not working: the sphinx.conf file had two "listen" lines, asking it to listen on two ports. I removed the second one (which probably should have been commented out) and now searchd is listening only to port 9312. Search has returned!
As stated above, basic searching works, but some of the other searches don't. Here's what I've found so far:
1) Profile posts cannot be searched; search defaults to an "everything" search. This one will need to have data indexed.
2) The "search posts and threads" tab has search options that are ignored, including searching against prefixes.
3) Member name search goes through Xenforo, and matches only on the exact name it seems. Is this worth converting this to Sphinx, or no? Doesn't vB offer wildcard or partial member name searches on the public side?
4) Cannot search within a single thread; search defaults again to an "everything" search.
#2 and #4 should not be too difficult to fix. #1 will need more work, and another index to make it happen. #3 is optional; it works, but could be converted to XF.
As stated above, basic searching works, but some of the other searches don't. Here's what I've found so far:
1) Profile posts cannot be searched; search defaults to an "everything" search. This one will need to have data indexed.
2) The "search posts and threads" tab has search options that are ignored, including searching against prefixes.
3) Member name search goes through Xenforo, and matches only on the exact name it seems. Is this worth converting this to Sphinx, or no? Doesn't vB offer wildcard or partial member name searches on the public side?
4) Cannot search within a single thread; search defaults again to an "everything" search.
#2 and #4 should not be too difficult to fix. #1 will need more work, and another index to make it happen. #3 is optional; it works, but could be converted to XF.
Wildcat Media
Well-known member
One tip: I updated my Sphinx on the sandbox forum to 2.0.5-release. In the distribution is a new sphinxapi.php file. I renamed it to "Sphinxapi.php" (as the filenames are case-sensitive) and replaced it in this plugin. The search still operates, albeit still broken in a few ways. But it is probably a good idea to include the proper API file to match the version of Sphinx you are using on your server.
As for searching profile posts, I did a few searches here and noticed that only profile posts are retrieved, not comments. That makes things easier.
As for searching profile posts, I did a few searches here and noticed that only profile posts are retrieved, not comments. That makes things easier.
DRE
Well-known member
When can you post an update of the addon? I've always wanted to see what the hype was about.One tip: I updated my Sphinx on the sandbox forum to 2.0.5-release. In the distribution is a new sphinxapi.php file. I renamed it to "Sphinxapi.php" (as the filenames are case-sensitive) and replaced it in this plugin. The search still operates, albeit still broken in a few ways. But it is probably a good idea to include the proper API file to match the version of Sphinx you are using on your server.
As for searching profile posts, I did a few searches here and noticed that only profile posts are retrieved, not comments. That makes things easier.
Wildcat Media
Well-known member
Not sure yet--I will need help coding parts of it. I am taking a shot right now at adding to sphinx.conf so it will index the profile posts. I am also now thinking that for the additional search capabilities, it will need some changes in the data being gathered for the post and thread indexes (for sorting and narrowing down the searches, thanks to the new search options in XF 1.1 and higher).
The basic functionality is already in place, and Sphinx works on maybe half of the search functions already in the forum. So I feel it is not that far away from being usable. But it needs that extra bit of a push to make it usable. I also struggle a bit since the coding style is different from my own.
Despite not having some of the features of ElasticSearch, I still feel it is a very valid search option for XF. A lot of us can't afford the type of managed server needed to afford us the memory for 7.5 million posts. Sphinx works fantastically well for us on vB3.7 straight from disk, which is why I can see its benefits on our upcoming XF move as well.
The basic functionality is already in place, and Sphinx works on maybe half of the search functions already in the forum. So I feel it is not that far away from being usable. But it needs that extra bit of a push to make it usable. I also struggle a bit since the coding style is different from my own.
Despite not having some of the features of ElasticSearch, I still feel it is a very valid search option for XF. A lot of us can't afford the type of managed server needed to afford us the memory for 7.5 million posts. Sphinx works fantastically well for us on vB3.7 straight from disk, which is why I can see its benefits on our upcoming XF move as well.
DRE
Well-known member
Yeah I can't afford all that stuff plus my board isn't a big board but I would still a search engine that works better than the default xenforo search engine. The current search engine frustrates me.
Wildcat Media
Well-known member
What is nice about Sphinx and ES is that you can change certain things, such as the stopwords list and minimum word length. Normally those are determined by MySQL and, if you are on a shared hosting account, those are parameters you don't ordinarily have access to.
I am looking at Sphinx now as a way to index other sites of mine that might grow larger. I'm working on two WordPress projects that could grow a bit (as both are "network" versions with multiple WP sites). Having Sphinx could really enhance searching in those sites.
Getting the data indexed and out of the XF tables is not hard at all. It's processing those indexes that is going to take me a bit of work (and what I'd need help doing). If I don't get any response here, I may see about getting paid help.
I am looking at Sphinx now as a way to index other sites of mine that might grow larger. I'm working on two WordPress projects that could grow a bit (as both are "network" versions with multiple WP sites). Having Sphinx could really enhance searching in those sites.
Getting the data indexed and out of the XF tables is not hard at all. It's processing those indexes that is going to take me a bit of work (and what I'd need help doing). If I don't get any response here, I may see about getting paid help.
Wildcat Media
Well-known member
Correction: it appears comments on profile posts are searched. I now see it on the search form. (Wonder why it didn't work here though? Bug?)
Now I have to rewrite what I was working on.
Now I have to rewrite what I was working on.
Similar threads
- Question
