XF 1.4 Problems importing attachments VB 4.2 to XF 1.4 - help!?

[gerryvz]

I'm doing a series of test imports of my existing VB4 forum to a virgin XF 1.4 forum + database to prepare for an eventual migration this fall.

Basic stats on my source VB forum: 4,800 users, 102,000 total posts, 7600 threads. Images and attachments are stored in the file system (not the database) and total around 12.8 GB of data. My hoster is 1and1 and the php memory limit is 120 MB. php 5.4.x is what they're using.

Everything is going seamlessly in terms of user, post, forum, PM migrations, etc. until the importation process gets to the "import attachments" step, when around 40% through, things fail with a serious error, as shown below.

My attachments and images are in the "images/attachments" folder of the forum (normal place in a VB install) and are nested below that in numbered folders, and include attachments + thumbs.

Can anyone detail what the issue with the attachment/image import process? I've researched the community via various other threads (shown below) and it seems that others have had similar problems in the past. I've tried a couple of the fixes (per the config.php file changes, for instance) and have not had luck in solving the issue.

Thoughts? Comments? Help? ?!?

Thanks,
Gerry

XenForo Community threads that seem relevant to my issue:

• Fatal error while importing attachments
• XF 1.4 - Import Attached Files Error
• XF 1.4 - Importing Images Failed Help
• Memory error during attachment import

Here is the error information that I'm getting about 43% of the way through the attachment import process:

Fatal error: Out of memory (allocated 82051072) (tried to allocate 2448 bytes) in /homepages/0/d315437492/htdocs/testforum2/library/XenForo/Image/Gd.php on line 215

Server Error

Mysqli statement execute error : Duplicate entry '18376' for key 'PRIMARY'

Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 574
Zend_Db_Adapter_Abstract->insert() in XenForo/DataWriter.php at line 1624
XenForo_DataWriter->_insert() in XenForo/DataWriter.php at line 1613
XenForo_DataWriter->_save() in XenForo/DataWriter.php at line 1405
XenForo_DataWriter->save() in XenForo/Model/Import.php at line 1890
XenForo_Model_Import->importAttachment() in XenForo/Model/Import.php at line 1936
XenForo_Model_Import->importPostAttachment() in XenForo/Importer/vBulletin4x.php at line 114
XenForo_Importer_vBulletin4x->stepAttachments() in XenForo/Importer/Abstract.php at line 97
XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 135
XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /homepages/0/d315437492/htdocs/testforum2/admin.php at line 13

 

[gerryvz]

I just tried another import and the attachment step failed at 43.8%:

Fatal error: Out of memory (allocated 82051072) (tried to allocate 9792 bytes) in /homepages/0/d315437492/htdocs/testforum2/library/XenForo/Image/Gd.php on line 215

Server Error
Mysqli statement execute error : Duplicate entry '18376' for key 'PRIMARY'

1. Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
2. Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
3. Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 574
4. Zend_Db_Adapter_Abstract->insert() in XenForo/DataWriter.php at line 1624
5. XenForo_DataWriter->_insert() in XenForo/DataWriter.php at line 1613
6. XenForo_DataWriter->_save() in XenForo/DataWriter.php at line 1405
7. XenForo_DataWriter->save() in XenForo/Model/Import.php at line 1890
8. XenForo_Model_Import->importAttachment() in XenForo/Model/Import.php at line 1936
9. XenForo_Model_Import->importPostAttachment() in XenForo/Importer/vBulletin4x.php at line 114
10. XenForo_Importer_vBulletin4x->stepAttachments() in XenForo/Importer/Abstract.php at line 97
11. XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
12. XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 135
13. XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 347
14. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
15. XenForo_FrontController->run() in /homepages/0/d315437492/htdocs/testforum2/admin.php at line 13

 

[Jake Bunce]

The "Out of memory" error likely indicates that there is one very large image attachment that is consuming too much memory for image operations during the import.

Ask your host to increase or disable PHP's memory_limit.

You can also try changing the memory_limit yourself. These methods don't always work, but try adding this to your library/config.php file:

 ini_set('memory_limit', -1);

Or add this to a .htaccess file on your server (only works on Apache servers):

 php_value memory_limit -1

The second error message about a 'Duplicate entry' might be incidental to the memory_limit error. Basically fix the memory_limit error, reinstall your XF forum, run the import again from scratch.

 

[Mike]

"Out of memory" would generally indicate your memory_limit isn't being hit. Something else is limiting the amount of memory PHP can use.

You may be able to workaround this by limiting the maximum size of images that will be resized by adding a line such as this to config.php:

$config['maxImageResizePixelCount'] = 10000000;

 

[gerryvz]

Thank you both for the suggestions. I had already tried Jake's workaround (detailed in another post) to no avail, but I will try yours Mike and report back.

It's a real bummer because most all other aspects of the import have been just fine, just requiring the normal tweaking and fine-tuning, Even some usergroup-related permissions for specific forum access came through nicely.

Thanks again for the suggestions, and more soon.

 

[gerryvz]

Hmmm... I implemented both of your suggestions for my config.php file in the /library folder after installing the forum but before beginning the import.

Unfortunately it hung up again at the same point, 43.88% of the image/attachment import process.

Gave me this error:

Fatal error: Out of memory (allocated 82051072) (tried to allocate 2448 bytes) in /homepages/0/d315437492/htdocs/testforum1/library/XenForo/Image/Gd.php on line 215


I'll talk to my hoster about upping or removing my php memory limitation, but don't have too much faith they would do this. We'll see.

Any thoughts on what an appropriate memory limit would be?

 

[gerryvz]

From what I can see, my hoster explicitly states in their rules that they will not increase php memory_limits for customers.

I'm wondering if it makes sense to do this migration on my own system, locally, and then to transfer everything directly to my hosted site?

Are there any pros and cons to doing it locally?

As an alternative, is there a way I could incrementally perform the image importation process? Say, in stages rather than all at once?

Or do the rest of the steps of the import (which work fine) and then "come back" to do the image import?

Cheers,
Gerry

 

[Paul B]

You can't do incremental imports, so it needs to be done all at once before going live.

Doing it locally is an option but that could mean a lot of data to download and upload, as well as a database dump and restore.
Depending on the size of the site, that could be impractical.
You would also likely need shell access for the database operations.

Or you could edit/delete the offending image in the source database (or the copy you are importing from) and then manually upload it again once the import has completed.

 

[gerryvz]

Or you could edit/delete the offending image in the source database (or the copy you are importing from) and then manually upload it again once the import has completed.

Is there any way to tell where in the database / which image is causing the import process to hang up?

Or would I need to go into the file system and check file size for each image in the sub-folders in the /attachments directory?

 

[Paul B]

Is there any way to tell where in the database / which image is causing the import process to hang up?

You can check the attachment table to see what the last successfully imported image was and it should be the one after that (in your VB installation).

 

[Rob]

I'm wondering if it would make any difference if the images were stored in the database on vBulletin, as opposed to the filesystem.
I remember when I imported that I moved all images back into the database prior to import - I cant for the life of remember why as it was a few years back. Sorry, this probably doesnt help.

 

[gerryvz]

The XenForo install imported about 15,889 images out of about 41,900 in the source vB database. I wasn't really able to find the offending image(s) in the source database attachment table, although immediately before the import to XF choked there were a few large images, where the same image was imported by XF about 5-6 times right before it choked. I deleted this group of about 5-6 large images from the source database and then tried to re-import on a fresh/virgin XF install and database.

Still hung up at the same point - 43.8% of the image importation.

As the vB and XF database attachment tables have different structures (and completely different tables), it is difficult to see where things correspond between the databases. Just to make sure ... can someone comment as to which attachment table in the vB database that XenForo uses as source data, and also which attachment table that XF uses as the corresponding table.

My hoster confirmed that they won't expand the php memory_limit beyond what it's already at (120 MB).

Here's a screen capture of the end of the importation where it choked, in the "attachment_data" table. As you can see there's a variety of image sizes and nothing looks out of the ordinary.

I'm wondering if it would make any difference if the images were stored in the database on vBulletin, as opposed to the filesystem.
I remember when I imported that I moved all images back into the database prior to import - I cant for the life of remember why as it was a few years back. Sorry, this probably doesnt help.

I think this would be too much messing around and hassle with vB, no? It's a working forum and I don't want to have to do anything that would cause disruption to the forum.

Also, if I remember correctly, the reason I didn't go with storing images in the database in the first place is that my hoster has a 1 GB limitation on databases, and I have 13GB of images already (and even my vB database currently is 275 MB and growing about 1-2 MB a month....)

What I DON'T want to have to do is go into all the threads after the forum has been imported, and have to re-attach each image to each post...sort of having visions of having to do that and I can't imagine the time that would be required. Not seeing an immediate solution to this problem, though I have plenty of time to work on it.

 

[Jake Bunce]

Server Error
Mysqli statement execute error : Duplicate entry '18376' for key 'PRIMARY'

1. Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
2. Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
3. Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 574
4. Zend_Db_Adapter_Abstract->insert() in XenForo/DataWriter.php at line 1624
5. XenForo_DataWriter->_insert() in XenForo/DataWriter.php at line 1613
6. XenForo_DataWriter->_save() in XenForo/DataWriter.php at line 1405
7. XenForo_DataWriter->save() in XenForo/Model/Import.php at line 1890
8. XenForo_Model_Import->importAttachment() in XenForo/Model/Import.php at line 1936
9. XenForo_Model_Import->importPostAttachment() in XenForo/Importer/vBulletin4x.php at line 114
10. XenForo_Importer_vBulletin4x->stepAttachments() in XenForo/Importer/Abstract.php at line 97
11. XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
12. XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 135
13. XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 347
14. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
15. XenForo_FrontController->run() in /homepages/0/d315437492/htdocs/testforum2/admin.php at line 13

This may indicate the attachment_id of the guilty attachment (18376) which, if you enabled the option to preserve the source ids, will be the same id in your vB database. Removing that attachment may avoid the problem.

Actually fixing the problem in your case may require running the import on a different server without memory constraints.

 

[Mike]

I'm still 99% sure the config.php option should sort it. (10000000 should be low enough, but it's possible it does need to be a little lower). The dupe ID error is likely only spawning from the other error, so it's a bit of a red herring.

If you can submit a ticket with access to the control panel and FTP access, we can look into it to see if we can get the import to complete. (Though I recognize you'll probably end up doing it again; hopefully the config.php will be all that's needed, though maybe with a lower number.)

 

[gerryvz]

OK, I'll try the config.php again, with a couple of different levels.

Just to be sure I'm doing things correctly, I should do a virgin forum+db install, THEN modify the config.php file in the /library subdirectory, and THEN begin the import process.

Correct?

Here's the (modified) config.php file I used from a prior import, that failed. Does it look OK ... everything in proper order? (XXXX is my crossing out a few digits/characters)


<?php

$config['db']['host'] = 'db58277XXXX.db.1and1.com';
$config['db']['port'] = '3306';
$config['db']['username'] = 'dbo58277XXXX';
$config['db']['password'] = 'XXXXXXXXX';
$config['db']['dbname'] = 'db58277XXXX';


$config['superAdmins'] = '1';

$config['maxImageResizePixelCount'] = 10000000;

ini_set('memory_limit', -1);

 

[Paul B]

Yes, that's the correct sequence and the config.php file looks OK but as Mike said, you may want to reduce the 10000000 to something lower to see if that allows it to pass.

 

[gerryvz]

Will do and report back. If it doesn't work at a couple of levels (I'll go a couple of different notches smaller), I have a fresh install+db instance ready to roll that I will use for a support ticket submission.

Appreciate the continued assistance and suggestions.

 

[gerryvz]

I'm still 99% sure the config.php option should sort it. (10000000 should be low enough, but it's possible it does need to be a little lower).

Yes, that's the correct sequence and the config.php file looks OK but as Mike said, you may want to reduce the 10000000 to something lower to see if that allows it to pass.

I went with 6000000 and 8000000 on the options, and seemed to hang up at the same place.

I'm also doing two test imports right now at the 15000000 and 25000000 levels, to see how things go if I put a higher value in.

I'll do one more test import at 8000000 and if that doesn't work, then I'll submit a ticket with private info.

More soon,
Cheers,
Gerry

 

[gerryvz]

Well, unfortunately the higher numbered config.php files also failed the install, so I'm a bit stuck for now. What a bummer !!

Seems to consistently hang up at the same place.

I'll submit a ticket with a virgin (no data) XF install with new database and provide the passwords and logins for you guys.

 

[gerryvz]

To close this issue out, it has been successfully resolved with the assistance of Mike.

Indeed, the problem was caused by a large image in the database, that had to be manipulated by XenForo for proper orientation/display, which temporarily duplicated the image and ate up double the memory that it typically would. The process of doing this caused the image to eat up all available memory, causing the error shown above.

The solution is to insert a line in the config.php file that limits the megapixel size of attachments/images that are imported from the DB.

In my case, given the memory limitations imposed by my hoster 1and1.com (different hosters have different levels for this) I used a 6000000 limitation number.

Thus, the line that needed to be inserted at the end of the php file is as follows:

$config['maxImageResizePixelCount'] = 6000000;

If anyone comes up with this problem in the future, this is the resolution for it. You can also experiment with different numbers (larger or smaller) for test imports. Smaller number if the import continues to fail with an "out of memory" error.

Kudos to Mike, Jake and Brogan for their assistance with this. I hope it's helpful for others.

Cheers,
Gerry