MG 1.1 Photopost import stays at 5.04% 1529 medias

[Triops]

After a successful (final test) migration of vBulletin 3.7 (with lots of attachments) I just run the PhotoPost - vBulletin import. First two points went well again, but media import hangs at 5.04% 1529 media (65k media to import). This is the second time, exactly at the same point. Stopping, re-running does not work.

But: Page refreshes as if it were perfectly importing,console confirms steady calls to admin.php?import/import, but no now media will beimported, it stays at 5.04% - second time now (imported backup after first time).

No nginx or sql error - what could it be please? How do I find out?
Current versions xf1.5.15 and mg 1.1.14

 

[Slavik]

Generally, when it keeps refreshing without continuing it is a particular image causing PHP to crash and then restart on a batch of photos.

If you check the gallery data folders are the same images being imported repeatedly?

 

[Triops]

Thank you. It stays always with 1529 medias in MySQL database.
Media Gallery Index shows different images, so no duplicates.

But in fmp we foubnd these logs:

[29-Sep-2017 09:49:39] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:39] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:39] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:40] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:40] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:40] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:40] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:41] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:41] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:41] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:41] WARNING: [pool forum] child 18695 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:41] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:49:42] WARNING: [pool forum] child 18696 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"

Would it make a difference to use GD2/ImageMagick to "jump" over wrong media file extension or similar?

 

[Triops]

When it first stopped, the children were not always the same:

[29-Sep-2017 09:03:59] WARNING: [pool forum] child 18484 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:03:59] WARNING: [pool forum] child 18484 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:03:59] WARNING: [pool forum] child 18461 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:03:59] WARNING: [pool forum] child 18461 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:00] WARNING: [pool forum] child 18485 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:00] WARNING: [pool forum] child 18485 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:00] WARNING: [pool forum] child 18484 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:01] WARNING: [pool forum] child 18484 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:01] WARNING: [pool forum] child 18461 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:01] WARNING: [pool forum] child 18461 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:01] WARNING: [pool forum] child 18485 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:04:02] WARNING: [pool forum] child 18485 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:05:56] WARNING: [pool forum] server reached max_children setting (4), consider raising it
[29-Sep-2017 09:34:32] WARNING: [pool forum] child 18582 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"
[29-Sep-2017 09:34:32] WARNING: [pool forum] child 18582 said into stderr: "Not a JPEG file: starts with 0x42 0x4d"

 

[Slavik]

You can try a different image processor, or, after taking a backup, remove the next media item set to be imported from the vbulletin database.

Alternatively you could look to hack the importer code to skip the problem image.

 

[Triops]

Will the setting for image processing within option "Attachments" from Xenforo core influence the image processing in media gallery too?

 

[Chris D]

It does for some operations, but not others. The image processing in XFMG is a bit disjointed - something we've remedied in XF2.

 

[Triops]

But ok, how can I change the image processor for importing PhotoPost, please? At the moment we stuck with the import, it won't proceed

This problem, with the same images to import, did not occur on a local XAMP 7.0.8 on Win10 64x (don't know, if that uses GD2 or IM)

 

[Chris D]

The import will have been with GD regardless of whether it happened on your server or a local server, and there's no particular way I can recommend of changing that..

Aside from the recommendations already given (essentially skipping the problematic item/s) I can only recommend working out what the other differences are between your local and server environment. Sometimes upgrading/recompiling PHP can help.

 

[Triops]

It worked now - since your importer keeps the origin media_ids I could identify, how far it has come. It was a bmp-bitmap from year 2004, but with ending .jpg. The same happened with another image of the user few percent later. Now it's at 22% percent already

Why this went through on XAMP I don't know....

 

[Alpha1]

I encountered the same issue on importing. I had to fix 5 images to complete the import.

 

[Triops]

Thank you @Alfa1, good to know that it is not as special misconfiguration on my AWS servers. I would assume, it's just a diffrent behaviour between the GD2 library on Win10 and that for Linux.

But I suffer another problem. After around 50% importing of 65 -75k images (have to check) the EC2/RDS instance become extremely slow. Now it takes 20-30 sec. for just 0.01% of progress. We checked CPU and memory, it's all fine and below 10%. Don't know, stopping the import and resuming 5 minutes later just continues to be slow... Don't know for the moment, where to check - it will take +12 hours to finish the last 25%... Something is wrong there...

 

[Alpha1]

I actually encountered something similar. This too was a bad file. It turned out to be malicious code in the file. I think it's best to find the file and inspect the size and content.
Over the decades it's inevitable to collect some bad files.

Maybe it's something similar for you.

 

[Triops]

Thanks again. One file is slowing down the whole import process? Shouldn't it speed up again after a faulty image is passed?

For me it's continously running and moving forward, but veeeery slow. After increasing disc space from 80GB to 250GB (to have more swap space if needed) it may have speed up a tiny bit, but it's still 0.01% within 15 secs....

 

[Alpha1]

I see. No what I experienced was the server taking ages the process one file. This sounds as something different.

I can really recommend using a server with high cpu because the import process is single threaded. So having a lot of cores is useless for import. This was an important factor why I was able to cut down the total import time (including All add-ons) from 7 days to just over 24 hours.

I did the import on a separate server and once complete I copied it all to the live server.

 

[Triops]

A lot to learn with AWS!
Your instance is not a own server, like used with my own IBM hardware. So the image import burned the "credits" I had for my EC2 instance -> CPU power was cut down almost totally... For next (final) migration I will book a large instance.