How fast should a vb3 import be? Can we speed it up?

HWS said:
And regarding to raw import time, everything above 6 hours is not acceptable.
Click to expand...

HWS, I love your bold statements.
First you talk about 7 days downtime, then you say that 7 hours wouldn't be acceptable.
:D

Rudy, search for his posts and you will find the reason of his negativity.
 
vB 3.8.7 to XF

Our ~225,000 posts ~30,000 threads, ~8,000 members and ~3.5 G of attachments took around an hour from start to finish.

I was quite surprised after hearing about how long this could take.
 
Further along, and now the mysql server has gone away. (Like Elvis, it has left the building.) I'm not sure what is timing out now.

Code: 
Mysqli prepare error: MySQL server has gone away
 
Zend_Db_Statement_Mysqli->_prepare() in Zend/Db/Statement.php at line 115
Zend_Db_Statement->__construct() in Zend/Db/Adapter/Mysqli.php at line 381
Zend_Db_Adapter_Mysqli->prepare() in Zend/Db/Adapter/Abstract.php at line 478
Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 734
Zend_Db_Adapter_Abstract->fetchAll() in XenForo/Importer/vBulletin.php at line 2237
XenForo_Importer_vBulletin->stepThreads() in XenForo/Importer/Abstract.php at line 77
XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 180
XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 132
XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 310
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 132
XenForo_FrontController->run() in /usr/www/users/xxxxxxxxx/admin.php at line 13
 
melbo said:
vB 3.8.7 to XF

Our ~225,000 posts ~30,000 threads, ~8,000 members and ~3.5 G of attachments took around an hour from start to finish.

I was quite surprised after hearing about how long this could take.
Click to expand...

What are your server specs? This test forum I'm converting is not faring all that well right now, and it is of a similar size with as many posts and threads (but only a few dozen users--the staff uses it for testing and forum business).

I really feel it is the database. Plus we are now hitting our peak hours on the main forum, and I'm stuck with an 80% complete import...
 
Rudy said:
And now that it is up around 72%, it is only importing about 0.4% per each page reload. :confused:
Click to expand...

The reported percentage isn't strictly speaking accurate. The importer doesn't do a row count to report the correct percentage. Rather it divides the autoincrement value of the current batch by the autoincrement value of the table. So, for example, if your threadids go up to 50000 but the first 10000 ids don't exist due to a previous prune of your forum then you will see the percentage jump to 20% at the beginning of the thread module, after which it will "slow down." In short, the rate at which the percentage progresses will vary depending on gaps in your autoincrement values.
 
We were starting from a completely empty XF install, Jake. I had a feeling it might not be accurate, but it seems like it is incrementally slower. Now with the server having "gone away," I'm having to babysit it and restart the import each time it bails on me. I only see one process in mytop from the queries, so it's not like we're running out of connections.​
​
Any recommendation on what I can safely bump the post "limit" to? I tried it at 200, then reduced it to 100--the database is still faltering.​
 
Walter said:
Rudy, search for his posts and you will find the reason of his negativity.
Click to expand...

There is NO "negativity". The truth is: Xenforo is simply not useable for large forums out of the box in it's current version. If we had known that 1.2 (with all the promised, currently missing but needed admin functions) will be "delayed" we would have never switched.

This is why I recommend not to switch to Xenforo currently without making massive customizations to it's code, if you own a large board.
 
Does anyone know if the importer performs extended inserts into MySQL? That is one reason for an error--there may just be too much data going to mysqld at any one time.

This is getting too frustrating. I imported the same forum a month ago as a test, and didn't hit this error. Nothing on the server has changed, other than another month's worth of posts. From the queries hitting the database, I am around post number 542000, yet our highest post number is around 680000.
 
HWS said:
There is NO "negativity". The truth is: Xenforo is simply not useable for large forums out of the box in it's current version. If we had known that 1.2 (with all the promised, currently missing but needed admin functions) will be "delayed" we would have never switched.

This is why I recommend not to switch to Xenforo currently without making massive customizations to it's code, if you own a large board.
Click to expand...

What specific complaints? Because I would very much disagree with your claim that XF is not usable for large forums. Quite the contrary I would say.
 
Jake Bunce said:
That error is caused by one of these MySQL settings being exceeded:

wait_timeout
max_allowed_packet

Your host or server person needs to increase these settings in MySQL. There is more info here:

http://dev.mysql.com/doc/refman/5.1/en/gone-away.html
Click to expand...

I had that thought myself, that it was wait_timeout. Since our loads were picking up when I attempted the import, mysqld was probably lagging and that is where wait_timeout came into play.

I did the import at 12:30am last night and it completed properly. Low traffic, no problem. Looks like the posts/threads imported in about 2-1/2 hours. 194,000 posts. (I was way off on the post count obviously, but there was some unusual lineage of the vB forum it came from. Long story.)

BTW, somewhat related--are there any MySQL tweaking threads like they used to have at "the other place"? George (eva2000) used to help me out all the time with recommended my.inf settings, further tweaking what I'd used, and it often helped. I trusted his word since he knew best how vB accessed the database, and which recommend settings would help us the most.
 
But still, that brings up the issue: if it took 2-1/2 hours to do 194,000 posts, that does not bode well for an import from 7.5 million posts. And the PMs ate up their fair share of time also. I bumped the limits on the thread/post import up like you recommended though.
 
Rudy said:
Looks like the posts/threads imported in about 2-1/2 hours. 194,000 posts. (I was way off on the post count obviously, but there was some unusual lineage of the vB forum it came from. Long story.)
Click to expand...

That module reports the thread count, not the post count.
 
I'm looking at the newly generated message count on the home page after the import, not in the importer. :) We have over 12,000 threads IOW.
 
You must log in or register to reply here.

Similar threads

AntonyF
  • Question Question
XF 1.2 Import speed - how long should it take, and can I make it faster?
Replies
7
Views
1K
AntonyF
AntonyF
Top Bottom