1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 Importing attachments fails on a regular basis

Discussion in 'Installation, Upgrade, and Import Support' started by Floren, Feb 27, 2012.

  1. Floren

    Floren Well-Known Member

    I recently purchased another Xenforo license and I'm in the process of converting a small 1mil posts vBulletin forum. The posts import worked flawless, where I have major issues is the attachments. Basically, every 5-10min the import fails. I have to refresh the page in order to restart the import.
    I'm running on Axivo MariaDB 5.2.10 and PHP 5.3.10 performance RPM's.

    What I noticed rightaway is that the import of attachments uses only one worker:

    stats.png

    That is because there is only one request sent at the time... but look at the huge CPU usage. Usually the load is distributed gradually through all workers, which makes this step a real performance issue. I'm not worried about the PHP size limitations or the number of open files, sysctl and other env settings are adjusted properly with plenty of room to spare for top performance. Any idea what can cause the random import failure? Is there a way I can run the import through console?

    I'm looking at 150GB worth of attachments... this looks like is going to be a very long import.
    Thanks for your help.
     
  2. Mike

    Mike XenForo Developer Staff Member

    If you're struggling to get it to finish, you may wish to decrease the number of attachments handled per page by editing the importer code. (Though you may need to restart the step entirely to ensure that change is applied.)

    This is a very server intensive step as it does a lot of image manipulation. If you wish, you could try Imagick instead of GD, though I don't know if this would help much.
     
  3. Floren

    Floren Well-Known Member

    Thanks for the good tip, Mike. I'm actually doing a test import now, so it does not matters that much if things fail. I'm sort of testing the waters on what steps need to be processed, setting permissions etc. So when the real import is done, I have a proper schedule with all steps documented. I will follow your advice and lower the batch number of attachments.
    I'm running already on php-pecl-imagick RPM, I've built that package as soon as I got Xenforo a while ago:
    Code:
    # yum --enablerepo=axivo list php-pecl-imagick
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    * base: mirrors.rit.edu
    * extras: centos.mirror.choopa.net
    * updates: mirrors.rit.edu
    Installed Packages
    php-pecl-imagick.x86_64              3.0.1-1.el5              axivo
    Scott did an awesome job with the PECL extension. I did not passed any recent tests but from previous experiences I believe that ImageMagick performs better than GD.
     
  4. Mike

    Mike XenForo Developer Staff Member

    Imagick may actually be a source of problems (with animated images), so trying GD is worth a shot too.
     
  5. Floren

    Floren Well-Known Member

    There are no animated images, all attachments are large pictures (max 2MB limit) of various products posted by users. Should I still try GD instead? I did not change any attachment settings so I guess they are being resized to the stock Xen size (1MB)... which would explain the huge processing I guess. :D
     

Share This Page