Lack of interest Improve search indexing speed

This suggestion has been closed automatically because it did not receive enough votes over an extended period of time. If you wish to see this, please search for an open suggestion and, if you don't find any, post a new one.

Adam Howard

Well-known member
I understand that rebuilding the search indexing after importing is required. I also understand that the speed of this does in part depend on the server's configuration.

And I say in part, because I'm convinced that part of that also depends on the proceed coded in XenForo. There are other "programs" that also offer rebuilding search for their software and they seem to do it a little quicker.

For example, I'm doing an import right now with 300,000 post and its taking about over an hour so far. On the same server, using "someone else" it was done in fewer time and they had 500,000 post (that's 1/2 a million done in less time)
 
Upvote 0
This suggestion has been closed. Votes are no longer accepted.
When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.
 
When you are rebuilding, what is your page delay and how many Items to process per page?
I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.

0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.
 
It's a bit of the nature of Elastic Search unfortunately... It can't get data direct from the database like Sphinx does... so basically you have to load the query results into PHP, format each individual record for insertion, insert it, etc. Sphinx on the otherhand just queries the MySQL database directly (no need to go through PHP).

I built a CLI indexer for XenForo that will index in chunks of 1,000 per node (meaning if you had 8 servers acting as Elastic Search nodes [which I do], you will get roughly 8x faster indexing speed). But if you are using a single Elastic Search node (which I think most people are), indexing via CLI really doesn't make any noticeable speed increase vs. doing it via web browser other than if you were having issues with the web indexing totally failing somehow.
 
0 second page delay
500 items per page

Still took well over an hour.

Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.

Maybe this is a trigger for customers to purchase Elastic Search :)
 
Maybe this is a trigger for customers to purchase Elastic Search :)

Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).
 
Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).

Don't you rebuild the search index after you import (at least with vB).
 
Don't you rebuild the search index after you import (at least with vB).

Yes, but you can't (shouldn't) install add-ons until you do (I wouldn't advise it).

But again, you're ignoring the fact that I brought up... Not everyone has VPS or Dedicated Servers, nor does everyone need one.

The search should be a little bit quicker for everyone and use just a fewer less resources.

If the competition can do it. I'm confident that so can XenForo. :)
 
Since the default search goes from a MySQL table to a MySQL table, you could probably speed it up 100x by doing the record formatting in the SQL query and just bypassing the PHP layer and just do an INSERT SELECT. Just an idea if someone was bored and wanted to try it.
 
Back
Top Bottom