1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Issue Search Index Rebuilding an Epic Fail

Discussion in 'Resolved Bug Reports' started by Floris, Oct 23, 2010.

  1. Floris

    Floris Guest

    I've got it running live, and it's simply impossible for me to rebuild the search index.
    I was kind of hoping beta 2 would have this improved, but it's no different.

    [​IMG]

    That's after 5 minutes .. I try to run it once a day, so it slowly gets more and more.

    But this means two third of my content can't be found via search or profiles after the 700k import.

    It's the only thing inside XenForo that's going horribly slow and causes the server to go on vacation.

    Can I perhaps run the query directly from MySQL (or will that have the same problems)
     
  2. Walter

    Walter Well-Known Member

    You are not utilizing much of your memory - have you tried to give mysql more RAM?
     
  3. hescominsoon

    hescominsoon Active Member

    i'd like to see a regular TOP to see your i/o wait. Your ram usage is very limited. give php, and mysql more ram. I bet you are going nuts with i/o wait since it appears your ram isn't being used hardly at all.

    Also under apache are you using prefork or worker mpm?
     
  4. Mike

    Mike XenForo Developer Staff Member

    I have made some changes here - mostly around allowing delays to be inserted - that will hopefully at least make it less of a server destroyer. But there is only so much that can be done, as most of the work/issues will be down to the underlying MySQL full text engine.
     
  5. Floris

    Floris Guest

    I appreciate the feedback guys, I will look into MySQL again, the info says it has already three times more than default, not sure why it's not utilizing it.

    Mike: thank you. Looking forward to beta 3.
     
  6. Floris

    Floris Guest

    The load is lower now, still hits 3 but it's not 13 after few minutes.
    But I do run into a lot more "An error occurred or the request was stopped."
     
  7. hescominsoon

    hescominsoon Active Member

    what are your php execution timeout settings set at?
     
  8. hescominsoon

    hescominsoon Active Member

    put in a batch command where only x are done at once. That way it stays within most php execution limits.
     
  9. Floris

    Floris Guest

    Double from the default.
     
  10. hescominsoon

    hescominsoon Active Member

    defaults are different control panel to control panel and distro to distro..what are the actual settings?
     
  11. hescominsoon

    hescominsoon Active Member

    I think adding an option into xf where you can specify the number of records per time in the acp would be a good idea here..that way the admins can manage the server load by doing the rebuild in batches...:)
     
  12. p4guru

    p4guru Well-Known Member

    I'd jack up the memory buffers in mysql temporarily just for rebuilding stage. What mysql version you using ?
     
  13. Floris

    Floris Guest

    Would it even benefit me if the php page (which one?) gets updated with this before it goes to the next page? adding a delay to it so i can just let it run in the browser and request it to do 250 or 500 or 750 per page. so the server has half a minute to catch up and lower load, before pushing it again with new stuff?

    sleep(30);
     
  14. hescominsoon

    hescominsoon Active Member

    yeah and code in something like a 5 second pause before it launches the next one. I would increase that until the php pause allows the system to catch up on the batch. I would not go higher than 500 at a time.
     
  15. Vladislav Rastrusny

    Vladislav Rastrusny Active Member

    Can you make options in control panel say

    • Maximum server load for intensive operations" (say, 6.0)
    • Pause to insert between load intensive operation steps if violating maximum server load
     
  16. Floris

    Floris Guest

    I don't know who set the title to [Fixed]
    But it very much so is not fixed.

    This feature is, despite the option to add a delay, still an epic fail.

    Most useless feature to me in XenForo.

    I am giving up on it. Cutting my losses.

    60 seconds delay per page:
    load average: 11.46, 8.67, 6.01

    After 7 hours it hits the 300k where it was, and spend another long time getting to 331k, and the server load goes up to stupid again.

    Must just be my luck I guess.
     
  17. hescominsoon

    hescominsoon Active Member

    how any is it trying to do at once? It sounds like it's trying to do too many too quickly. I don't know the environment on your machine intimately but VB can be easily tuned inside the UI to NOT have this thing happen. Kier we need something similar to be able to reign in the usage of XF on the machine during a rebuild.
     
  18. hescominsoon

    hescominsoon Active Member

    Floris maybe it's fixed in an upcoming beta?
     
  19. Floris

    Floris Guest

    Can the [fixed] be removed from the thread title, it's utterly false.

    [​IMG]

    I do not know why I have such bad luck with this, I even paid $50 the other month to try it on a linode vps for beta 5 and just tried beta 6 again on linode vps and leaseweb dedicated

    And it's a PoS performance, killing the server.

    One could say I have yet to complete the convert from vBulletin to XenForo because of this, data is missing on the profiles of people who are looking for their content just because of these stupid FULLTEXT indexed data and the way XenForo uses it "smarter".

    An utter disappointment; a horrible experience; greatly frustrating; delaying running a 750k site properly because of it.

    Please accept your product isn't supreme and address this before going stable. If you want real big customers converting away from vBulletin this is going to prevent them from having a ten year old millions of posts community converted to a great software like xenforo.

    Post history is important.
     
    hescominsoon likes this.
  20. hescominsoon

    hescominsoon Active Member

    wow. I wish i had a larger site to test this on. I'm curious if using fulltext is a good idea? What storage engine is being used myisam or inno?
     

Share This Page