Confirmed vBulletin Blog importer gets 'stuck' importing blog comments when using parallel importer

[36degrees]

Affected version

v2.1.10 Patch 2, XFI 1.3.2

Full disclosure upfront: I'm trying to import blogs from vBulletin 3.6, so I've hacked together my own importer and blog trait that extend the existing vBulletin 3.7 / 3.8 importer. The only difference is that blog tags do not exist in 3.6, so that step is removed in my trait's getSteps function. I believe this issue would exist with the stock 3.7/3.8 importer, but unfortunately I do not have access to a vB3.7 forum with blogs to test.

When importing using the CLI with multiple processes, the import gets stuck on the blog comments step. Once the blog comments have been migrated (you can see them in the database), the child processes just keep running in the background. The reporter gets stuck on:

Step 18 of 18: Blog comments             00:00:01 [1,234] 0.00%

If you kill the importer and re-run it without the --processes argument, the import will then finish successfully. (If you run it again with the processes argument it just gets stuck again).

In case it's relevant, there were 3,703 blog comments to migrate and I was running it with 6 processes.

 

[HLR]

This exact same thing happened to me today.

I'm doing a trial import of a vB4 forum on RDS into XF (on a different RDS instance) with 5.6m posts and 12,900 blog comments. I was running with 4 processors in CLI import mode, and it got stuck on Blog Comments. Every time I would ctrl-c and restart the import, the same record count would be added to the existing one. So if I tried 5 times, the total record count in the CLI importer would be 64.500.

The first time I did the trial import was in single proc mode and had no problems.

I did exactly what you did, restart with just one processor and it cleaned through and finished the import. I have no idea where those 5x blog comment records went, but it doesn't matter, this was just a trial migration anyway.

I'm now re-running the import with a t2.xlarge EC2 instance and 8 processors, and it's 5-6 times faster than the single threaded import. I may try bumping that up to 16 procs and see what happens.

 

[Ivancas]

I was running with 4 processors in CLI import mode

how do you run multiple processors in CLI import mode?

 

[HLR]

When using the CLI you can add "--help" to get a full list of options available.

Right now I'm re-running my migration with 32 processors:

 php cmd.php xf:import --processes=32 -vvv --verbose

Althought the -vvv and --verbose options don't seem to change anything.

But --processes=32 makes the import run super-fast on an 8-vCPU AWS instance.

I'm going to try again with 48 procs next. I wonder how high I can go before it starts to bog down.

The first time I did the import in single processor mode, it took 29 hours just to import 5.6million posts.
With 32 procs, it looks like that will be about 1 hour and 12 minutes.

 

[HLR]

Before you start your import, there's a "CLI Importer" tab you can click for info.