XF 1.1 Rebuilding Cache failed after vb3.8 import

[Mutt]

My install was successful & my 23 million post import finally completed. The cache has been rebuilding for most of the day but got stuck here

----------------------------------------
Rebuilding Caches...

Rebuilding... Threads 96,248
----------------------------------------

I tried hitting refresh to get it started again but received this message

----------------------------------------
An error occurred or the request was stopped.
----------------------------------------

I have 2 buttons I can click. the "Continue Rebuilding" button brings me to a blank page w/ nothing happening. The "Stop Rebuilding" brings me to a the rebuild is complete page but I seriously do not think it's over.

I checked the Server Error Logs & there's nothing from today.


does anyone have any advice on how to get this bad boy going again? It appears the I can click the "stop rebuilding" button & then head over to "Tools / Rebuild Caches" & get it restarted from there but is there something different I should be trying?

thanks in advance
Mutt

 

[Mutt]

the server seems fine, the drive is not full. I hit back a couple times & the cache starting to rebuild again but got hung up & gave me the error again in about the same spot.

in case it matters, I checked the DB & it's not what I expected it to be. larger than the VB DB i'm importing but less records. I guess I could have done a double import or something. not sure if this is normal. for example, here are 2 before & after post tables
vb posts - 22,478,295 records MyISAM latin1_swedish_ci 6.4 GiB
xen posts - 17,899,246 records InnoDB utf8_general_ci 12.1 GiB

 

[Dan]

Is it stopping at the exact same point every time? I know you said "about", but if it is the exact spot, that could possibly say something.

 

[Mutt]

That's just it, it wasn't exact. It was 96284 then I backed up & it gave me the error around 91000. I got it going again & that time it reached 96371 before giving me the error. I stopped messing w/ it then worrying that maybe I was making things worse.

I rain to an error while importing but that was caused by the hard drive being full. I deleted a ton of oversized logs & was able to get it running again. I went and checked again to see if I had bloated logs but there weren't.

 

[Jake Bunce]

the "Continue Rebuilding" button brings me to a blank page w/ nothing happening.

A blank page is often a suppressed error. Try adding this line to your library/config.php file:

ini_set('display_errors', true);

Or add this to a .htaccess file on your server (only works on Apache servers):

php_value display_errors 1

That may reveal an error message on the blank page.

 

[Mutt]

Jake, thanks so much. i'll see what that turns up.

 

[Mutt]

after making that change, here's the error I'm getting
------------------
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 39 bytes) in /var/www/virtual/sternfannetwork.com/public_html/xen/library/Zend/Db/Statement/Mysqli.php on line 298
------------------

 

[Paul B]

XenForo sets the memory limit to 128 MB, try increasing it in config.php:

ini_set('memory_limit', 256 * 1024 * 1024);

Or set it to -1 to remove the limit altogether.

 

[Mike]

He's unlikely to be able to change the limit based on that error, as it should already be at least 128M. The change will have to happen in php.ini.

 

[Mutt]

XenForo sets the memory limit to 128 MB, try increasing it in config.php:

ini_set('memory_limit', 256 * 1024 * 1024);

Or set it to -1 to remove the limit altogether.

I made this change & it's going again.!! if it craps out again I'll try making the change in php.ini

 

[Mutt]

i made the change to php.ini & rebooted. it's working now. 256M wasn't enough but 512M seems to have done the trick. at least for now.

 

[Mutt]

it says it finished. I'm still concerned about what I'm seeing in phpmyadmin. thread records are only slightly decreased so maybe we lost 3,500 threads out of 588,000. not too worried about that. but what the hell happened to the posts? it looks like 1/2 the posts (11 million) are gone but for some reason the 1/2 half that made it are taking up twice the space as the original instead of 1/2. certainly seems broken. even if the import was merging posts to account for the smaller size, why the larger file. and if the larger file was cause by threads getting imported twice, why the smaller record count? i'm totally confused on this one.

BEFORE IMPORT
thread - 588,700 records - MyISAM - latin1_swedish_ci - 179.2 MiB
post - 22,478,295 records - MyISAM - latin1_swedish_ci - 6.4 GiB

AFTER IMPORT
xf_thread - 585,111 records - InnoDB - utf8_general_ci - 137.2 MiB
xf_post - 11,855,347 records - InnoDB - utf8_general_ci - 12.1 GiB

 

[Mutt]

next up, should I do the search index now or am I wasting my time until I get the posts straightened out? or is the upcoming big board search around the corner & won't require it?

 

[Mike]

You can't compare sizes between InnoDB and MyISAM. InnoDB takes more space in general.

As for the counts, as I noted, you can't trust an InnoDB count in phpMyAdmin/MySQL metadata unless you actually run a (very slow) query to do a proper count. Alternatively, you had orphan data that wasn't imported. I'd look around for missing data.

MySQL full text search really won't be usable with that much data. I probably wouldn't even bother indexing the old content unless it's going somewhere else.

 

[Mutt]

thanks mike.. I wasn't sure how I'd find missing post among millions that made it but if this isn't that unusual, I'll just move on.

I was thinking i'd install apache locally & redo the import here to see if I got the same results. Should I even bother?

and thanks about the search index. Is there anyway to limit the searxh index to just the last years worth of posts or something like that where the newest stuff would still be searchable?

 

[Jake Bunce]

Regarding row counts:

http://xenforo.com/community/threads/phpmyadmin-row-count-changes-every-time-i-browse-xf_post.14180/

Run that query to get the correct row count.

and thanks about the search index. Is there anyway to limit the searxh index to just the last years worth of posts or something like that where the newest stuff would still be searchable?

That is not a configurable option. It requires modifying the code.

 

[Mutt]

Regarding row counts:

http://xenforo.com/community/threads/phpmyadmin-row-count-changes-every-time-i-browse-xf_post.14180/

Run that query to get the correct row count.

i ran it in phpmyadmin & it's been running for what seems like hours