• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Lack of interest Improve search indexing speed

Adam Howard

Well-known member
#1
I understand that rebuilding the search indexing after importing is required. I also understand that the speed of this does in part depend on the server's configuration.

And I say in part, because I'm convinced that part of that also depends on the proceed coded in XenForo. There are other "programs" that also offer rebuilding search for their software and they seem to do it a little quicker.

For example, I'm doing an import right now with 300,000 post and its taking about over an hour so far. On the same server, using "someone else" it was done in fewer time and they had 500,000 post (that's 1/2 a million done in less time)
 

Moshe1010

Well-known member
#2
When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.
 

Adam Howard

Well-known member
#3
When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.
0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.
 

digitalpoint

Well-known member
#4
It's a bit of the nature of Elastic Search unfortunately... It can't get data direct from the database like Sphinx does... so basically you have to load the query results into PHP, format each individual record for insertion, insert it, etc. Sphinx on the otherhand just queries the MySQL database directly (no need to go through PHP).

I built a CLI indexer for XenForo that will index in chunks of 1,000 per node (meaning if you had 8 servers acting as Elastic Search nodes [which I do], you will get roughly 8x faster indexing speed). But if you are using a single Elastic Search node (which I think most people are), indexing via CLI really doesn't make any noticeable speed increase vs. doing it via web browser other than if you were having issues with the web indexing totally failing somehow.
 

Moshe1010

Well-known member
#7
0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.
Maybe this is a trigger for customers to purchase Elastic Search :)
 

Adam Howard

Well-known member
#9
Maybe this is a trigger for customers to purchase Elastic Search :)
Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).
 

Adam Howard

Well-known member
#11
Don't you rebuild the search index after you import (at least with vB).
Yes, but you can't (shouldn't) install add-ons until you do (I wouldn't advise it).

But again, you're ignoring the fact that I brought up... Not everyone has VPS or Dedicated Servers, nor does everyone need one.

The search should be a little bit quicker for everyone and use just a fewer less resources.

If the competition can do it. I'm confident that so can XenForo. :)
 

digitalpoint

Well-known member
#13
Since the default search goes from a MySQL table to a MySQL table, you could probably speed it up 100x by doing the record formatting in the SQL query and just bypassing the PHP layer and just do an INSERT SELECT. Just an idea if someone was bored and wanted to try it.