1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Cache rebuilds are incredibly slow...

Discussion in 'Installation, Upgrade, and Import Support' started by Ryan McAdams, Dec 6, 2010.

  1. Ryan McAdams

    Ryan McAdams Active Member

    So I don't want to be "that guy" who complains but I have to share my experience....

    I've just finished a fairly large import, 150k users, a few million posts and I kicked off the cache reubild and it's incredibly slow.... for large sites this is going to be a problem... now on my smaller sites this is no big deal but this literally is going to take well over 24 hours for a full conversion.

    Maybe I'm crazy but it seems like there should be some kind of out-of-browser option for larger sites to help with this?
     
  2. Jesepi

    Jesepi Well-Known Member

    Like a shell script or something?
     
  3. Ryan McAdams

    Ryan McAdams Active Member

    Exactly...

    I've been through this three times now... and getting the fatal error trying to allocate 8 bytes... I've bumped the php memory limit to 256 megs to see if it'll make it through this time... but it would be nice to be able to do imports and cache rebuilds through something like a bash script ;)
     
  4. Ryan McAdams

    Ryan McAdams Active Member

    I've talked to a few people who agree on this...

    Mike/Kier is there any chance that one of you guys could even possibly explain to us the architecture and what it's doing so we can write this on our own? Floris has gotten off to a good start but he's worried about his data not matching up with yours.

    Thoughts?
     
  5. Mike

    Mike XenForo Developer Staff Member

    The main logic for each rebuilder is in library/XenForo/CacheRebuilder. From there it spreads out, but you should be able to follow the logic through the classes.
     
  6. Floris

    Floris Guest

    So when i posted about this wirh 'epic fail' in the title i was not wrong, this feature is going to be a showstopper for bigboards
     
  7. Ryan McAdams

    Ryan McAdams Active Member

    Ya - I mean we're not a 'huge' board but this is a problem for us... 140k users and ~4M posts and this process is crushing us :(
     
  8. Floris

    Floris Guest

    I've had to do this on a single dedicated box for 750k and a 3 dedicated servers setup with apache2/php on one, and mysql on the other, (third one is backup and cdn) and each box had 2 quad core cpus and 8gb ram .. that one had a looooot more millions of posts. We couldn't complete it on any of them. Loads went through the roofs, even with beta 3's option to delay per page load. I've started writing a shell script but I just don't quite master the way xenforo's framework does things - results don't match up; I will revisit it all after 1.1.x and see what has improved. We need to focus on other elements of the site. And are using google search for content prior to convert.
     
  9. Ryan McAdams

    Ryan McAdams Active Member

    I actually though the problem could be my web server/db server combo so I pulled my database down local to my dev box here... It's a dual Quad Core, 32 gigs of RAM - loaded up my DB with a default install of Ubuntu Server (requirements met) and pulled in my database...

    I still (with no traffic and virtually unlimited resources) cannot get a search cache rebuild to complete...
     
  10. Floris

    Floris Guest

    Beta5 seems to have made things worse. I can't even complete it now: It runs, and within 2 minutes:

    [​IMG]
    Request state:

    array(3) { ["url"] => string(53) "http://wetalknation.net/admin.php?tools/cache-rebuild" ["_GET"] => array(1) { ["tools/cache-rebuild"] => string(0) "" } ["_POST"] => array(5) { ["process"] => string(1) "1" ["caches"] => string(136) "[["SearchIndex",{"batch":500,"start":122,"extra_data":{"content_types":["thread"],"current_type":"post","type_start":61000},"delay":0}]]" ["position"] => string(1) "1" ["redirect"] => string(47) "http://wetalknation.net/admin.php?tools/rebuild" ["_xfToken"] => string(53) "1,1292667724,<removed>" } }
     
  11. Mike

    Mike XenForo Developer Staff Member

    Yeah, that's actually a bug in ZF (actually, a bug caused by a bug fix for ZF). I've worked around it for the next release.
     
  12. Floris

    Floris Guest

    Thanks Mike for the info.
     
  13. MGSteve

    MGSteve Well-Known Member

    Is the workaround postable, I'm going to be facing this problem shortly as well.
     

Share This Page