1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Lack of Interest Improve search indexing speed

Discussion in 'Closed Suggestions' started by Adam Howard, Jun 17, 2013.

  1. Adam Howard

    Adam Howard Well-Known Member

    I understand that rebuilding the search indexing after importing is required. I also understand that the speed of this does in part depend on the server's configuration.

    And I say in part, because I'm convinced that part of that also depends on the proceed coded in XenForo. There are other "programs" that also offer rebuilding search for their software and they seem to do it a little quicker.

    For example, I'm doing an import right now with 300,000 post and its taking about over an hour so far. On the same server, using "someone else" it was done in fewer time and they had 500,000 post (that's 1/2 a million done in less time)
  2. Moshe1010

    Moshe1010 Well-Known Member

    When you are rebuilding, what is your page delay and how many Items to process per page?
    I did 500K in about 10min or less. I set 0 sec page delay and 1000 items per page.
  3. Adam Howard

    Adam Howard Well-Known Member

    0 second page delay
    500 items per page

    Still took well over an hour.

    Installing elastic search for XenForo advance search made this process fast and it did it in 5 minutes... But respectfully, even without the elastic search, this should be faster.
  4. digitalpoint

    digitalpoint Well-Known Member

    It's a bit of the nature of Elastic Search unfortunately... It can't get data direct from the database like Sphinx does... so basically you have to load the query results into PHP, format each individual record for insertion, insert it, etc. Sphinx on the otherhand just queries the MySQL database directly (no need to go through PHP).

    I built a CLI indexer for XenForo that will index in chunks of 1,000 per node (meaning if you had 8 servers acting as Elastic Search nodes [which I do], you will get roughly 8x faster indexing speed). But if you are using a single Elastic Search node (which I think most people are), indexing via CLI really doesn't make any noticeable speed increase vs. doing it via web browser other than if you were having issues with the web indexing totally failing somehow.
  5. imno007

    imno007 Well-Known Member


    OP was talking about the Xenforo default search, not Elastic Search. ;)
  6. digitalpoint

    digitalpoint Well-Known Member

    Oh... Lol. Oops
    Adam Howard likes this.
  7. Moshe1010

    Moshe1010 Well-Known Member

    Maybe this is a trigger for customers to purchase Elastic Search :)
  8. Adam Howard

    Adam Howard Well-Known Member

    Don't feel bad. I've been skim reading things as of lately too. :p
  9. Adam Howard

    Adam Howard Well-Known Member

    Shouldn't be required and can't exactly be implemented in advance for imports (as it would add to the server load during an import).

    Further more, not everyone is using a VPS or Dedicated Server (nor should they be required).
  10. Moshe1010

    Moshe1010 Well-Known Member

    Don't you rebuild the search index after you import (at least with vB).
    SneakyDave likes this.
  11. Adam Howard

    Adam Howard Well-Known Member

    Yes, but you can't (shouldn't) install add-ons until you do (I wouldn't advise it).

    But again, you're ignoring the fact that I brought up... Not everyone has VPS or Dedicated Servers, nor does everyone need one.

    The search should be a little bit quicker for everyone and use just a fewer less resources.

    If the competition can do it. I'm confident that so can XenForo. :)
  12. Mike

    Mike XenForo Developer Staff Member

    SneakyDave likes this.
  13. digitalpoint

    digitalpoint Well-Known Member

    Since the default search goes from a MySQL table to a MySQL table, you could probably speed it up 100x by doing the record formatting in the SQL query and just bypassing the PHP layer and just do an INSERT SELECT. Just an idea if someone was bored and wanted to try it.
    Jarod and Adam Howard like this.

Share This Page