Sphinx Search Engine

[mlx]

Sphinx Search Engine v0.0.2 for XenForo v1.0.0 (gold/final/stable)

Please consider this a very first proof of concept beta release.

XenForo is new to me. So if I'm doing something terribly wrong here please let me know

The following Installation and Upgrade instructions assume that you already know how to setup Sphinx Search Engine, including how to setup cronjobs to reindex your search index and update your delta search index inbetween. See below for a more detailed howto for CentOS5 (including how to setup Sphinx itself).

I've been using the latest Sphinx 1.10-beta release. An example sphinx.conf is included.

I believe it should work with Sphinx 0.9.9 as well. I didn't have time to test this though.


Quick Installation

1. Upload all files from the "upload" folder to your XenForo directory
2. Go to your XenForo Admin Control Panel and click Install Add-on
3. Select addon_xt_sphinx.xml and click Install Add-on
4. add the source and index definitions from the included sphinx.conf to your sphinx.conf
5. reindex your search index
6. enjoy your Sphinx Search Engine
7. please report any bugs and suggestions

Quick Upgrade

1. Upload all files from the "upload" folder to your XenForo directory
2. Go to your XenForo Admin Control Panel and click List Add-ons
3. Look for Sphinx Search Engine 0.0.1 and click Controls -> Upgrade
4. Select addon_xt_sphinx.xml and click Upgrade Add-on
5. enjoy your Sphinx Search Engine
6. please note you no longer need to manually edit
   library/XenForo/Search/SourceHandler/Abstract.php
7. please report any bugs and suggestions

Detailed HowTo (for CentOS5)
(off the top of my head)

Download and compile Sphinx

•      # wget http://sphinxsearch.com/files/sphinx-1.10-beta.tar.gz
       # tar xzvf sphinx-1.10-beta.tar.gz
       # cd sphinx-1.10-beta
       # ./configure --prefix=/usr/local/sphinxbeta
       # make
       # make install
       # useradd -M searchd
       # chown searchd.searchd /usr/local/sphinxbeta/var/data
       # chown searchd.searchd /usr/local/sphinxbeta/var/log

Setup your Sphinx config

•       # cd /usr/local/sphinxbeta/etc
        # vi sphinx.conf

• press

  i

• paste the content from examples/sphinx.conf
• press

  [esc] :wq [enter]

Setup PHP scripts

• Upload all files from the "upload" folder to your XenForo directory
• Go to your XenForo Admin Control Panel and click Install Add-on
• Select addon_xt_sphinx.xml and click Install Add-on

Setup startup script

•        # cd /etc/init.d
         # vi searchd

• press

  i

• paste the content from examples/searchd_initd
• press

  [esc] :wq [enter]

•        # chmod +x searchd
         # mkdir /var/run/sphinxbeta
         # chown searchd.searchd /var/run/sphinxbeta
         # /bin/su -c "/usr/local/sphinxbeta/bin/indexer --all" searchd
         # service searchd start

Setup cron scripts

•        # cd /etc/cron.daily
         # vi sphinx_reindex

• press

  i

• paste the content from examples/sphinx_reindex
• press

  [esc] :wq [enter]

•        # chmod +x sphinx_reindex
         # ./sphinx_reindex
         # cd /etc
         # mkdir cron.1minute
         # vi crontab

• press

  i

• add this

  * * * * * root run-parts /etc/cron.1minute

• press

  [esc] :wq [enter]

•        # cd cron.1minute
         # vi sphinx_delta

• press

  i

• paste the content from examples/sphinx_delta
• press

  [esc] :wq [enter]

•        # chmod +x sphinx_delta
         # ./sphinx_delta

Make sure searchd is up on reboot

•        # chkconfig --add searchd
         # chkconfig searchd on

P.S. personally I'm reindexing every 24 hours at 4a.m. and run the delta each and every minute. This setup works fine for us, but we only have about 3 million posts to reindex anyway. So for really big forums a delta update every 5 minutes might be better (that would be */5 * * * * root run-parts /etc/cron.5minutes in crontab, in the howto above). And you need to decide when and if to run the reindex. That really depends on your setup and hardware I guess.

 

[Shadab]

Just a quick note; nothing serious:
The variable $db is undefined in SphinxSearch_Install::uninstallSphinx().

Nice work, btw. I wish my forum(s) required Sphinx search.

 

[mlx]

D'oh, you are so right.

Edit: I updated the ZIP file. I didn't update the version number for this tiny little bug. I guess the worst thing it does is not dropping the 4 sphinx tables when you decide to uninstall.

 

[Andy.N]

Looking forward to your beginner guide.
I hope your first step would help make the sphinx integration into default xenforo

 

[ExpertPixels.com]

"sphinx integration" yes it is tricky so a user-friendly / beginner guide would be great, i screwed up a vb4 installation attempting this lol so i will wait for a full guide also. Thanks

 

[RvG]

Just a quick note; nothing serious:
The variable $db is undefined in SphinxSearch_Install::uninstallSphinx().

Nice work, btw. I wish my forum(s) required Sphinx search.

a brand new software with sphinx installed?

 

[Blandt]

a brand new software with sphinx installed?

Some are porting old cranky vB 3's with gazillions of posts

 

[Anthony Parsons]

I wish my forum(s) required Sphinx search.

Yer... that would be awesome to have a massive forum with over a million posts. Nice work...

 

[jonsidneyb]

I only have half a million posts but I like that this is out here.

 

[Weiyan]

Have brothers and sisters tested this mod in Chiinese envirornment? Pls. share your experience.

 

[mlx]

The Add-On should be fine.

It's a matter of your sphinx.conf though.

The included example uses a default charset_table that supports English and German.

See http://www.sphinxsearch.com/wiki/doku.php?id=charset_tables for other languages.

Also see ngram_len and ngram_chars for CJK (Chinese, Japanse, Koreasn) support for unsegmented texts.

 

[jonsidneyb]

At what size does Sphinx start becoming a meaningful addition?

 

[Andy.N]

shouldn't this be useful at any size? so if you have a small site, will this hurt?

 

[Forsaken]

shouldn't this be useful at any size? so if you have a small site, will this hurt?

Smaller sites usually are on shared hosting.

 

[Blandt]

Smaller sites usually are on shared hosting.

I think what Andy means small as in less than a million post

In Sphinx terms a million post is a small tiny site

 

[Andy.N]

Ok, shared hosting issue aside, what kind of site that sphinx would be ideal for?
I'm always of the thought that extra power, better code, optimized search engine can only help in the long run.

 

[jonsidneyb]

I am at about 500,000 now so well below a million. Well, half that many.

 

[Andy.N]

i love to see this built into Xenforo at some point.

 

[Forsaken]

i love to see this built into Xenforo at some point.

http://xenforo.com/community/threads/xenforo-running-with-278-000-posts.334/page-2#post-56793
http://xenforo.com/community/threads/technology-used-by-xenforo.7/#post-24
http://xenforo.com/community/thread...cached-alternative-web-servers.431/#post-4576

 

[mlx]

On our test import on a test server XenForo MySQL FT search queries took up to 10 seconds. Compared to <1s with Sphinx. So I guess that's when Sphinx starts making sense. When your MySQL FT search slows your site down. If it's responsive enough you might just want to stick to it.