XF 1.1 Importing attachments fails on a regular basis

[Floren]

I recently purchased another Xenforo license and I'm in the process of converting a small 1mil posts vBulletin forum. The posts import worked flawless, where I have major issues is the attachments. Basically, every 5-10min the import fails. I have to refresh the page in order to restart the import.
I'm running on Axivo MariaDB 5.2.10 and PHP 5.3.10 performance RPM's.

What I noticed rightaway is that the import of attachments uses only one worker:



That is because there is only one request sent at the time... but look at the huge CPU usage. Usually the load is distributed gradually through all workers, which makes this step a real performance issue. I'm not worried about the PHP size limitations or the number of open files, sysctl and other env settings are adjusted properly with plenty of room to spare for top performance. Any idea what can cause the random import failure? Is there a way I can run the import through console?

I'm looking at 150GB worth of attachments... this looks like is going to be a very long import.
Thanks for your help.

 

[Mike]

If you're struggling to get it to finish, you may wish to decrease the number of attachments handled per page by editing the importer code. (Though you may need to restart the step entirely to ensure that change is applied.)

This is a very server intensive step as it does a lot of image manipulation. If you wish, you could try Imagick instead of GD, though I don't know if this would help much.

 

[Floren]

If you're struggling to get it to finish, you may wish to decrease the number of attachments handled per page by editing the importer code.

Thanks for the good tip, Mike. I'm actually doing a test import now, so it does not matters that much if things fail. I'm sort of testing the waters on what steps need to be processed, setting permissions etc. So when the real import is done, I have a proper schedule with all steps documented. I will follow your advice and lower the batch number of attachments.

This is a very server intensive step as it does a lot of image manipulation. If you wish, you could try Imagick instead of GD, though I don't know if this would help much.

I'm running already on php-pecl-imagick RPM, I've built that package as soon as I got Xenforo a while ago:

# yum --enablerepo=axivo list php-pecl-imagick
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.rit.edu
* extras: centos.mirror.choopa.net
* updates: mirrors.rit.edu
Installed Packages
php-pecl-imagick.x86_64              3.0.1-1.el5              axivo

Scott did an awesome job with the PECL extension. I did not passed any recent tests but from previous experiences I believe that ImageMagick performs better than GD.

 

[Mike]

Imagick may actually be a source of problems (with animated images), so trying GD is worth a shot too.

 

[Floren]

There are no animated images, all attachments are large pictures (max 2MB limit) of various products posted by users. Should I still try GD instead? I did not change any attachment settings so I guess they are being resized to the stock Xen size (1MB)... which would explain the huge processing I guess.