XF 2.2 About importing huge vbulletin board to xenforo 2.x

[Eran]

Hey Guys!
We are willing to migrate our site to xenforo.
We have 250m posts, 80m private messages, etc... the DB is very big.
I tried to use the importer but it takes two weeks minimum (with the CLI option) to import it.

My question is simple: If I won't retain the ID, Can I open the forum for our members parallel with the big imports? posts, threads, messages... Or do I have to wait for the importer to be finished before any activity?
There are better options to make the importer faster? I tried to update the limit variable with no success. Our server can handle more than 500 records in raw. How can I do that?

Also, Do you know/provide any importing services for sites that big? to reduce the import time from 2 and half weeks to a few hours?

Thanks, guys!
Eran

 

[rdn]

There are better options to make the importer faster?

Rent a temporary, powerful server for the import; it should only take a few hours.
Like:

Dedicated Root Server Hosting

Find your reliable dream server, optimized for high capacity use, ever-growing storage needs or rapid and parallel processing.

www.hetzner.com

Dedicated Server Hosting

Find your reliable dream server, optimized for high capacity use, ever-growing storage needs or rapid and parallel processing.

www.hetzner.com

 

[benFF]

Rent a beefy "by the minute" cloud server and use it just to do the import, if you don't have many attachments then it can do it super fast.

I was able to to import 5m posts in about 20 minutes.

Attachments take a lot longer as they can't go in parallel.

 

[Miri]

Amazing! Just like importing a whole continent. Glad you chose Xenforo for your forum.

 

[Miri]

Or do I have to wait for the importer to be finished before any activity?

It depends on the traffic of your users, but personally, I would opt for this solution. I'm curious to know what experienced users on this forum think or recommend.

 

[Paul B]

If I won't retain the ID, Can I open the forum for our members parallel with the big imports? posts, threads, messages... Or do I have to wait for the importer to be finished before any activity?

Do not open the site until the import is finished.

As per the previous posts, a a sufficiently sized and configured server will be able to bring the time down considerably.

 

[MySiteGuy]

A bigger server will help, but due to some of the table scans in the import process, 250 million posts are not going to scale to the same number of rows imported per hour as a forum with a few million (or even tens of millions). It won't be days, but it certainly won't be just a few hours. Threads with large numbers of replies can be especially long to import. The database also needs to be optimized for high write loads - out of the box MariaDB/MySQL have write bottlenecks even with the fastest NVMe drives.

This is a perfect use case for incremental import. Import the site while it's live, without bringing it down. Then work on testing and validating the import while the original site stays live. After you've validated and tested to your satisfaction, then you do an incremental import. I do this all the time. Only the incremental import stage needs the live forum turned off and that's a relatively short period of time.

 

[digitalpoint]

So this doesn't help you with XF2, but "back in the day" I made my own importer that effectively bypassed XenForo's system and used MySQL's LOAD DATA method (much faster for bulk loading data into tables).

You may want to consider something similar if you are trying to do 100s of millions of records.

This won't work for you with XF2 (it's from 10 years ago for XF1), but you can get an idea of how long it took:

A.R.F.I (A Really Fast [vB4] Importer)

Disclaimer: This is listed as $100, but it's free for premium members of our website (which costs $25).[/B] This is the CLI-based importer that we custom built to retain all primary keys and import EVERYTHING from vBulletin 4 to XenForo 1.1.3...

xenforo.com

Grand Total For Everything: 33 minutes, 3 seconds (74,587,823 records)

 

[Eran]

Rent a temporary, powerful server for the import; it should only take a few hours.
Like:

Dedicated Root Server Hosting

Find your reliable dream server, optimized for high capacity use, ever-growing storage needs or rapid and parallel processing.

www.hetzner.com

Dedicated Server Hosting

Find your reliable dream server, optimized for high capacity use, ever-growing storage needs or rapid and parallel processing.

www.hetzner.com

Rent a beefy "by the minute" cloud server and use it just to do the import, if you don't have many attachments then it can do it super fast.

I was able to to import 5m posts in about 20 minutes.

Attachments take a lot longer as they can't go in parallel.

It depends on the traffic of your users, but personally, I would opt for this solution. I'm curious to know what experienced users on this forum think or recommend.

Do not open the site until the import is finished.

As per the previous posts, a a sufficiently sized and configured server will be able to bring the time down considerably.

A bigger server will help, but due to some of the table scans in the import process, 250 million posts are not going to scale to the same number of rows imported per hour as a forum with a few million (or even tens of millions). It won't be days, but it certainly won't be just a few hours. Threads with large numbers of replies can be especially long to import. The database also needs to be optimized for high write loads - out of the box MariaDB/MySQL have write bottlenecks even with the fastest NVMe drives.

This is a perfect use case for incremental import. Import the site while it's live, without bringing it down. Then work on testing and validating the import while the original site stays live. After you've validated and tested to your satisfaction, then you do an incremental import. I do this all the time. Only the incremental import stage needs the live forum turned off and that's a relatively short period of time.

So this doesn't help you with XF2, but "back in the day" I made my own importer that effectively bypassed XenForo's system and used MySQL's LOAD DATA method (much faster for bulk loading data into tables).

You may want to consider something similar if you are trying to do 100s of millions of records.

This won't work for you with XF2 (it's from 10 years ago for XF1), but you can get an idea of how long it took:

A.R.F.I (A Really Fast [vB4] Importer)

Disclaimer: This is listed as $100, but it's free for premium members of our website (which costs $25).[/B] This is the CLI-based importer that we custom built to retain all primary keys and import EVERYTHING from vBulletin 4 to XenForo 1.1.3...

xenforo.com

Thank you, guys!
about getting a super server, we currently have strong 2 dedicated servers, but the import task runs only one process that does not take even 9% of the CPU.
I saw the xf:import supporting parallel import, but it's always on one process. What do I missing?
I tried to run

nohup nice -20 php cmd.php xf:import > importer_logs.txt &

Tried to allocate defined cores, tried to increase php memory, sql pool memory, linux process resources allocation, but nothing make it faster.

-There's any additional server side configuration that I might missed or anything that can help?
-How do I use the parallel import option?
-My vbulletin original is 3.x but it was upgraded to 4.2.x back then. I might need to choose vbulletin 3.x importer? that might be the issue?
-Is there perhaps a way to optimize the vbulletin DB to make the importer faster?


Thank you guys.

 

[Wildcat Media]

So this doesn't help you with XF2, but "back in the day" I made my own importer that effectively bypassed XenForo's system and used MySQL's LOAD DATA method (much faster for bulk loading data into tables).

I remember reading about that when you released it--still impressive!

 

[benFF]

What do I missing?

You're missing the command line parameter to tell it to use multiple processes

--processes X

 

[Eran]

You're missing the command line parameter to tell it to use multiple processes

--processes X

You're right, I ended up finding it finally yesterday, but when it comes to threads or private messages - MySQL DeadLock message. Any suggestion to prevent this error with no reduced process number?
thanks!