XF 2.1 Import from VB4.x fails during attachments

[IbrahimIMC]

So I'm doing a fairly large board import from VB to XF and have tried twice, running from the CLI, only to have the import fail in a non-recoverable state during the attachments import.

I imagine there is a file that it's dying at, but it's not giving filenames and I don't know of any logfile I can look at to see the issue.

This is the last line before it just fails:

- Step 19 of 26: Attachments 00:45:19 [98,447] 19.52%

Running the import from the CLI again will start as usual, try to resume from the attachments, and just drop back to the command line with no message, no error, no time elapsed.

Any ideas? (Or, better yet, point me to where error logging can be turned on or found?)

Thanks!

 

[Paul B]

You can try running the command again and adding --verbose, which may provide more details.

Otherwise, the last attachment imported into XF is the one just before the one which is failing (in the VB database).

 

[IbrahimIMC]

Running from the CLI with verbose turned on didn't give me any more info. But I think I found the file that comes after the last successful file. It's named a653ae1a13c2e40093a010fd7784c1a4.gif and has an 'attachmentid' in the VB database one increment higher than the last attachment found in the XF database.

Nothing seems odd about the entry, however. So what should I do now? Should I attempt to drop rows and see if it can push on through or...?

 

[Paul B]

Are you able to upload that attachment here?

You can try deleting that attachment from the file system and the importer should be able to get past it, but it would be interesting to see why it's failing.

 

[Mike]

If the CLI process is just ending, that seems almost certainly like PHP is crashing. This is something we see occasionally, most commonly with attachments. Unfortunately, because of how it happens though, it's essentially impossible to predict or account for. (Just opening the image for thumbnailing, for example, can be enough to kill PHP.)

Generally, you'd want to skip importing the file. The easiest way to do this is to remove the record from the source database or, if it points a file on the disk, rename that file (which should cause us to skip it).

 

[Chris D]

With it being a gif though I’d be curious as to whether it’s particularly large (either in dimensions of file size), and whether you have XF configured to use Imagick or the default GD for processing images.

There’s a theoretical issue if Imagick is enabled and we’re dealing with a gif that has lots of frames. Usually issues arising from that would manifest themselves differently (out of memory errors) but it could be an explanation.

 

[IbrahimIMC]

So an update: I'm going to run the importer from blank again. This time everything except the attachments, and then run the attachments separately.

I do want to see what the status is of everything outside of the attachments first. I wasn't able to find the .gif on the disk, though it's supposed to be stored there.

I had a few warnings during the Avatar import that were all variations of:

libpng warning: iCCP: known incorrect sRGB profile
libpng warning: Interlace handling should be turned on when using png_read_image

The warnings didn't stop the import, and so I don't think it's a problem. But posting it here just for more info.

I'll let you know what happens next.

 

[Mike]

Those warnings are mostly normal if you run via the CLI. They are output by a very low level library used by PHP and can't be suppressed.

 

[IbrahimIMC]

Well, we got rid of one failed attachment and it chugged along for a while before crashing out again. Here is the error in the db log:

*** 794. row ***
error_id: 794
exception_date: 1549495770
user_id: 0
ip_address:
exception_type: ErrorException
message: Fatal Error: imagecreatefromjpeg(): gd-jpeg: JPEG library
reports unrecoverable error: Unsupported marker type 0x44
filename: src/XF/Image/Gd.php
line: 40
trace_string: #0 [internal function]: XF::handleFatalError()
#1 {main}
request_state: a:1:{s:3:"cli";s:27:"cmd.php xf:import --verbose";}

Not really sure on the what or why of this error, but maybe this will help find a solution?

 

[Mike]

Unfortunately that is essentially the same thing and actually shows why we can't do anything about it: it's actually a fatal error, so it terminates PHP when it's triggered. You'd need to follow the same process to remove the file from the source.

Alternatively, you if you have the PECL imagick extension installed, you may be able to switch XF's image processing to use that. It may handle corrupted images better than the GD default. If I'm correct in thinking that this is PHP 5.6, using a more recent version may also help due to underlying bug fixes.

 

[JamesAus]

Those warnings are mostly normal if you run via the CLI. They are output by a very low level library used by PHP and can't be suppressed.

Is this still current and is a known issue and doesn't cause problems? I'm getting it with an import.

 

[IbrahimIMC]

So the problem was definitely in corrupted uploads.
We fixed the problem by writing a shell script to basically run the code that xf imports on every attachment we had. Every time it got the fatal error, it wrote the filename down in a log file for us. Once we had that, we deleted all the files. The import was able to run after that.