XF 2.2 vB5 to XF2.2.4, running importer 1.4.1 in docker

[VincentD]

Hi all,

I'm running an import with importer 1.4.1 in a docker container setup to migrate a vBulletin forum. The vBulletin SQL dump is around 900MB to give an indication of the size. I have the following setup to migrate:

version: '2'

services:
  php:
    image: xfrocks/xenforo:php-apache
    environment:
      - VIRTUAL_HOST=dev.local.airwork.nl
    ports:
      - '8100:80'
    links:
      - mysql
      - vbulletin
    volumes:
      - /Users/myuser/Documents/xenforo/html/:/var/www/html/

  mysql:
    image: mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: 'database'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password'
    ports:
      - '9906:3306'

  vbulletin:
    image: mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: 'vbulletin'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password'
    cap_add:
      - SYS_NICE # CAP_SYS_NICE
    ports:
      - '9907:3306'

The import is still running but very slowly. I'm running on a macbook pro and I started the importer from the CLI with the following command:

php cmd.php xf:import --processes=10

I've given the docker process more resources which did speed it up a little. (CPUs: 10, Memory: 16GB, Swap 4GB, Disk image size; 100GB)

Now for the import. It's currently on step 14 and already running for more then 150 hours.

Starting import from vBulletin 5.x...
Using parallel runner with up to 10 processes.
Note: completion percentage is approximate and only updated periodically.
 - Step 14 of 21: Posts                     153:40:24 [353,731] 0.12%

I don't see any errors in the docker logs and the import is still progressing. Anything else I can check to see how to speed things up?

 

[Kirby]

150 hours? jesus!

Smth. ist definitly wrong, I've run imports (from vBuleltin 4) with 10M+ posts within a couple hours.

Is MySQL/MariaDB configured properly? I'd expect that it is not which could explain why it is that insanely slow.

 

[VincentD]

150 hours? jesus!

Smth. ist definitly wrong, I've run imports (from vBuleltin 4) with 10M+ posts within a couple hours.

Is MySQL/MariaDB configured properly? I'd expect that it is not which could explain why it is that insanely slow.

AFAIK this is a standard config. To add to this. I've tried to optimize 2 tables from vbulletin (text and node) but that doesn't impact performance. I'll optimize all the tables and try again.

 

[MySiteGuy]

The out of the box MySQL and MariaDB configurations are horribly bad for any write intensive work such as imports.

 

[VincentD]

Thanks for replying. I'm trying to find out what a config for a write intensive mysql should be. I'm currently testing with adding the following to the mysql images and that seems to speed it up somewhat:

command: --innodb_buffer_pool_size=8096M --innodb_read_io_threads=8

 

[VincentD]

Actually, it might be a different issue.

There's only 63649 records in my xf_post table while there are 323150 rows in my (vbulletin) text table. The importer claims to be on 361001?

 

[VincentD]

I retried with a clean database, latest versions and storage pointing to a folder outside of docker but no luck. After 2 hours the import finished 8000PM's. Which should be done in matter of minutes (seconds). I'm going to try native although I wish I hadn't. Running it in docker is so much easier.

 

[VincentD]

A follow-up in case anyone comes across this thread. I managed to import my vbulletin 5 database. I stopped using docker on a macOS and I switched to my windows desktop. I installed XAMPP and restored my vbulletin database there. Then I started a new xenforo install and used the 1.5.0 importer. The import was done in 20 minutes! I did have to tinker with some settings to allow more resources for MySQL but other then that no trouble at all.