How fast should a vb3 import be? Can we speed it up?

Wildcat Media

Wildcat Media

Well-known member
I just converted a test vB 4.0 forum to XF 1.1. While importing threads/posts, it seems like the import started dragging at some point. For the latter part of that section of the import, the importing was taking about 60-80 seconds to convert about 500 posts (or IOW, about 0.25% progress every 60-80 seconds). For a couple of other functions, such as PMs, I noticed the same slowdowns--the first portion seemed fast, yet it got to a point where imports slowed to a crawl.

Total posts number almost 193,000 and total threads is just over 12,000. The entire import for posts/threads was 3 hours, 39 minutes.

Since I'm doing this in preparation to move a vB 3.7 big board (7.5 million posts), I'm wondering what the best method would be to import everything. I'd read of a command line interface here--is it that much faster without the overhead of HTTP? Or if I run in GUI mode, is there something I should do on the server beforehand? Would increasing the PHP memory limit for XF help? I don't want to be down for an entire week doing a conversion, IOW...it should go much faster than it was earlier. (BTW, I had done this test import before, and while I don't remember exact numbers, it certainly did not take this long.)

I was converting on the same server as the big board, so that would account for some of the delay. However, when we do the conversion, we will be running on the server with no other load on it. I also noticed during the import that our server loads and disk I/O did not change appreciably.
 
You're best to do an optimize and repair on MySQL before imporating, but generally speaking the default importer is a little slow doing it remotely.... Best to do this on a localhost if you can.
 
You might be able to improve performance by increasing the batch sizes for each import step. Edit this file:

library/XenForo/Importer/vBulletin.php

Search for 'limit' =>. You will find a limit defined for each import module.

I have found that this can help on very large imports, but only if your server has the resources to handle it. For example, too large of a batch size might hit certain server limits such as PHP's memory_limit.
 
I would prefer a command line importer--isn't there one kicking around for vB 3.7/3.8? I agree I would rather go direct to the shell/command line than deal with HTTP overhead, since we have 7.5 million posts to get through.
 
There is a command line one. Gotta import the XML:

library/XFCliImporter/addon-XFCliImporter.xml

But it's not technically supported, and I have seen people report problems with it.
 
Aha, so it's been right in front of me the whole time. :D Thanks!

I may test it out on another forum and see what happens. If not, I'll definitely increase the limits as you've suggested--I may be able to bump up PHP memory usage if I need it to finish a conversion.
 
Rudy said:
Aha, so it's been right in front of me the whole time. :D Thanks!

I may test it out on another forum and see what happens. If not, I'll definitely increase the limits as you've suggested--I may be able to bump up PHP memory usage if I need it to finish a conversion.
Click to expand...

Bumping it onto ramdisk has also proven quite effective at speeding things up
 
Slavik said:
Bumping it onto ramdisk has also proven quite effective at speeding things up
Click to expand...
I can setup a ramdisk on FreeBSD no problem, but what do I use the ramdisk for? And, how large should it be? Since our traffic will be nothing, we won't have to worry about a load other than the conversion script, so I can easily spare memory if needed.
 
vB4 import was understandable since the way the tables were set up, making imports hard to sit through.

For me, it was fast because I had somewhat big board with 2,200+ members (at the time).

But my vB3 import was surprisingly fast. I just tried it a week ago, I converted a 52k posts, and 4k members vb3 site to xenforo.

I was like "wow." :)
 
Slavik said:
Bumping it onto ramdisk has also proven quite effective at speeding things up
Click to expand...
I was just thinking about RAMdisk imports the other day. Has anyone tried this?

I imagine it would be fairly economical considering you can run high-performance cloud/virtual servers with >32GB RAM for less than $1.00/hour these days. Then just copy your new database back over to your production server and XF installation and you're done.
 
Short of having a 64GB Ramdisk and having both the new and old forum databases loaded into it, I don't see what else I'd use it for. Our database is at least five times the size of our available memory.

Great in theory but we don't have the resources to do it.
 
I'm doing a test import of vB4 right now.

One thing I'm noticing (and I think i mentioned it earlier in another thread, yet I can't find it) is that the import hits a brick wall and slows to a crawl. The PMs from vB4 imported quickly up until it got to just over 60%, and now it is taking ages.

My PHP memory limit is 128M. If I kick up the batch sizes as Jake mentioned, would this help make the rest of the import faster?

I'm following the database activity using mytop and notice that database activity is low. Averaging between 3 and 4 queries per second, and at times it looks like it is doing nothing. But, it only samples the database once per second, so I know it is not quite showing me everything.
 
I am betting this might be the problem:

vb 4 import: 504 Gateway Time-out on "profile comments" and "threads/posts"

When I moved from our old hosting company (which our previous admin used) to our new one, the database import from a dump file had taken forever. I found out later that it was the MySQL indexing that was holding it up, and that I could have temporarily removed the database keys (indexes) during the import of the dump file, then add the keys back in, which actually was more efficient (in a batch, vs. doing them one at a time).

Might it be good to remove the keys from the affected tables right before import, then add them back after?
 
Doing the posts/threads import right now. I bumped the limit for threads from 100 to 200, and the posts limit from 800 to 1200 (as we have no threads over 1000 posts).

The first batch imported about 27.5% of the posts and threads. Each successive batch is only biting off about 2.5% each time.

Sure smells like database congestion to me. When we do the Big Board, something needs to be done to speed the import up.
 
Expect a week downtime for a 7.5 million posts board.

We converted a 3.5 million board using a RAM disk (otherwise it simply wouldn't have worked out). We had 3 days "read-only" time at the old board and 1 day down time. The CLI importer is not relyable, if you don't want to loose many threads and posts during the import do not use it!

Also you should do all tweaking to posts and users BEFORE the import in vB. XF does not have a lot of admin features, so you simply cannot mess around with posts and users AFTER the import. There are simply NO mass administration utilities in XF currently (promised for XF 1.2, if it ever comes).

My recommendation would be to wait until XF 1.2 will be available before thinking about converting such a large board. If you do not want to wait, expect a lot of payment for custom coding. Even the import can be made drastically quicker with custom coding (ask digitalpoint).
 
Walter said:
Nowadays there is a command line importer (which is much faster) and I know more about Xenforo :)
Click to expand...

Yes, the CLI importer is faster. It also will loose about 1 million posts per 5 million posts imported. If you care about that, you won't use it. :)

And regarding to raw import time, everything above 6 hours is not acceptable.
 
You must log in or register to reply here.

Similar threads

AntonyF
  • Question Question
XF 1.2 Import speed - how long should it take, and can I make it faster?
Replies
7
Views
1K
AntonyF
AntonyF
Back
Top Bottom