Cache rebuilds are incredibly slow...

[Ryan McAdams]

So I don't want to be "that guy" who complains but I have to share my experience....

I've just finished a fairly large import, 150k users, a few million posts and I kicked off the cache reubild and it's incredibly slow.... for large sites this is going to be a problem... now on my smaller sites this is no big deal but this literally is going to take well over 24 hours for a full conversion.

Maybe I'm crazy but it seems like there should be some kind of out-of-browser option for larger sites to help with this?

 

[Jesepi]

Like a shell script or something?

 

[Ryan McAdams]

Like a shell script or something?

Exactly...

I've been through this three times now... and getting the fatal error trying to allocate 8 bytes... I've bumped the php memory limit to 256 megs to see if it'll make it through this time... but it would be nice to be able to do imports and cache rebuilds through something like a bash script

 

[Ryan McAdams]

I've talked to a few people who agree on this...

Mike/Kier is there any chance that one of you guys could even possibly explain to us the architecture and what it's doing so we can write this on our own? Floris has gotten off to a good start but he's worried about his data not matching up with yours.

Thoughts?

 

[Mike]

The main logic for each rebuilder is in library/XenForo/CacheRebuilder. From there it spreads out, but you should be able to follow the logic through the classes.

 

[Floris]

So when i posted about this wirh 'epic fail' in the title i was not wrong, this feature is going to be a showstopper for bigboards

 

[Ryan McAdams]

Ya - I mean we're not a 'huge' board but this is a problem for us... 140k users and ~4M posts and this process is crushing us

 

[Floris]

I've had to do this on a single dedicated box for 750k and a 3 dedicated servers setup with apache2/php on one, and mysql on the other, (third one is backup and cdn) and each box had 2 quad core cpus and 8gb ram .. that one had a looooot more millions of posts. We couldn't complete it on any of them. Loads went through the roofs, even with beta 3's option to delay per page load. I've started writing a shell script but I just don't quite master the way xenforo's framework does things - results don't match up; I will revisit it all after 1.1.x and see what has improved. We need to focus on other elements of the site. And are using google search for content prior to convert.

 

[Ryan McAdams]

I actually though the problem could be my web server/db server combo so I pulled my database down local to my dev box here... It's a dual Quad Core, 32 gigs of RAM - loaded up my DB with a default install of Ubuntu Server (requirements met) and pulled in my database...

I still (with no traffic and virtually unlimited resources) cannot get a search cache rebuild to complete...

 

[Floris]

Beta5 seems to have made things worse. I can't even complete it now: It runs, and within 2 minutes:

Request state:

array(3) { ["url"] => string(53) "http://wetalknation.net/admin.php?tools/cache-rebuild" ["_GET"] => array(1) { ["tools/cache-rebuild"] => string(0) "" } ["_POST"] => array(5) { ["process"] => string(1) "1" ["caches"] => string(136) "[["SearchIndex",{"batch":500,"start":122,"extra_data":{"content_types":["thread"],"current_type":"post","type_start":61000},"delay":0}]]" ["position"] => string(1) "1" ["redirect"] => string(47) "http://wetalknation.net/admin.php?tools/rebuild" ["_xfToken"] => string(53) "1,1292667724,<removed>" } }

 

[Mike]

Yeah, that's actually a bug in ZF (actually, a bug caused by a bug fix for ZF). I've worked around it for the next release.

 

[Floris]

Thanks Mike for the info.

 

[MGSteve]

Yeah, that's actually a bug in ZF (actually, a bug caused by a bug fix for ZF). I've worked around it for the next release.

Is the workaround postable, I'm going to be facing this problem shortly as well.