Lack of interest Improve search indexing speed

[Adam Howard]

I understand that rebuilding the search indexing after importing is required. I also understand that the speed of this does in part depend on the server's configuration.

And I say in part, because I'm convinced that part of that also depends on the proceed coded in XenForo. There are other "programs" that also offer rebuilding search for their software and they seem to do it a little quicker.

For example, I'm doing an import right now with 300,000 post and its taking about over an hour so far. On the same server, using "someone else" it was done in fewer time and they had 500,000 post (that's 1/2 a million done in less time)

 

[Moshe1010]

When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.

 

[Adam Howard]

When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.

0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.

 

[digitalpoint]

It's a bit of the nature of Elastic Search unfortunately... It can't get data direct from the database like Sphinx does... so basically you have to load the query results into PHP, format each individual record for insertion, insert it, etc. Sphinx on the otherhand just queries the MySQL database directly (no need to go through PHP).

I built a CLI indexer for XenForo that will index in chunks of 1,000 per node (meaning if you had 8 servers acting as Elastic Search nodes [which I do], you will get roughly 8x faster indexing speed). But if you are using a single Elastic Search node (which I think most people are), indexing via CLI really doesn't make any noticeable speed increase vs. doing it via web browser other than if you were having issues with the web indexing totally failing somehow.

 

[imno007]

@digitalpoint

OP was talking about the Xenforo default search, not Elastic Search.

 

[digitalpoint]

Oh... Lol. Oops

 

[Moshe1010]

0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.

Maybe this is a trigger for customers to purchase Elastic Search

 

[Adam Howard]

Oh... Lol. Oops

Don't feel bad. I've been skim reading things as of lately too.

 

[Adam Howard]

Maybe this is a trigger for customers to purchase Elastic Search

Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).

 

[Moshe1010]

Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).

Don't you rebuild the search index after you import (at least with vB).

 

[Adam Howard]

Don't you rebuild the search index after you import (at least with vB).

Yes, but you can't (shouldn't) install add-ons until you do (I wouldn't advise it).

But again, you're ignoring the fact that I brought up... Not everyone has VPS or Dedicated Servers, nor does everyone need one.

The search should be a little bit quicker for everyone and use just a fewer less resources.

If the competition can do it. I'm confident that so can XenForo.

 

[Mike]

Well, a larger per page will generally help. The most significant thing you can do - likely destroying any incremental changes - is to disable keys, build the index, and then enable them (http://stackoverflow.com/questions/8210608/mysql-disable-enable-keys).

 

[digitalpoint]

Since the default search goes from a MySQL table to a MySQL table, you could probably speed it up 100x by doing the record formatting in the SQL query and just bypassing the PHP layer and just do an INSERT SELECT. Just an idea if someone was bored and wanted to try it.