• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Cache rebuilds are incredibly slow...

#1
So I don't want to be "that guy" who complains but I have to share my experience....

I've just finished a fairly large import, 150k users, a few million posts and I kicked off the cache reubild and it's incredibly slow.... for large sites this is going to be a problem... now on my smaller sites this is no big deal but this literally is going to take well over 24 hours for a full conversion.

Maybe I'm crazy but it seems like there should be some kind of out-of-browser option for larger sites to help with this?
 
#3
Like a shell script or something?
Exactly...

I've been through this three times now... and getting the fatal error trying to allocate 8 bytes... I've bumped the php memory limit to 256 megs to see if it'll make it through this time... but it would be nice to be able to do imports and cache rebuilds through something like a bash script ;)
 
#4
I've talked to a few people who agree on this...

Mike/Kier is there any chance that one of you guys could even possibly explain to us the architecture and what it's doing so we can write this on our own? Floris has gotten off to a good start but he's worried about his data not matching up with yours.

Thoughts?
 

Mike

XenForo developer
Staff member
#5
The main logic for each rebuilder is in library/XenForo/CacheRebuilder. From there it spreads out, but you should be able to follow the logic through the classes.
 
F

Floris

Guest
#6
So when i posted about this wirh 'epic fail' in the title i was not wrong, this feature is going to be a showstopper for bigboards
 
F

Floris

Guest
#8
I've had to do this on a single dedicated box for 750k and a 3 dedicated servers setup with apache2/php on one, and mysql on the other, (third one is backup and cdn) and each box had 2 quad core cpus and 8gb ram .. that one had a looooot more millions of posts. We couldn't complete it on any of them. Loads went through the roofs, even with beta 3's option to delay per page load. I've started writing a shell script but I just don't quite master the way xenforo's framework does things - results don't match up; I will revisit it all after 1.1.x and see what has improved. We need to focus on other elements of the site. And are using google search for content prior to convert.
 
#9
I actually though the problem could be my web server/db server combo so I pulled my database down local to my dev box here... It's a dual Quad Core, 32 gigs of RAM - loaded up my DB with a default install of Ubuntu Server (requirements met) and pulled in my database...

I still (with no traffic and virtually unlimited resources) cannot get a search cache rebuild to complete...
 
F

Floris

Guest
#10
Beta5 seems to have made things worse. I can't even complete it now: It runs, and within 2 minutes:


Request state:

array(3) { ["url"] => string(53) "http://wetalknation.net/admin.php?tools/cache-rebuild" ["_GET"] => array(1) { ["tools/cache-rebuild"] => string(0) "" } ["_POST"] => array(5) { ["process"] => string(1) "1" ["caches"] => string(136) "[["SearchIndex",{"batch":500,"start":122,"extra_data":{"content_types":["thread"],"current_type":"post","type_start":61000},"delay":0}]]" ["position"] => string(1) "1" ["redirect"] => string(47) "http://wetalknation.net/admin.php?tools/rebuild" ["_xfToken"] => string(53) "1,1292667724,<removed>" } }
 

Mike

XenForo developer
Staff member
#11
Yeah, that's actually a bug in ZF (actually, a bug caused by a bug fix for ZF). I've worked around it for the next release.