Is there some sort of speed limit for importing?

[NolF]

I have a very large board (400k users, 1.7m posts) and the speed at which just users are being imported is eternally slow. Combine that with some duplicate entry errors and it will never finish :S

Is there a trick to make it run faster? I have a dedicated Q6600 offpeak and the load is very low (~1)....

 

[webroxau]

When testing this afternoon it took 1 hour 45 minutes to import 750000 posts alone without the time taken for all the other items.

 

[NolF]

Been playing with the importer open library\XenForo\Importer\Vbulletin.php

Changing the value from 100 to 1000 or more makes it considerably faster

'limit' => 100,

There are four entries, one for Visitor Meeseges, Users, Threads, Polls. Change it as desired

edit: ~2h for 750k posts, hopefully it won't take ~4h to import the ~2Gb worth of posts :/

 

[Mike]

It will take a long time to import that much data. There's only so much that can be done. Aside from the data formats being different, consider that all the data has to be converted to UTF-8 as well.

 

[NolF]

Fair points Mike, I hadn't considered the conversion Increasing the limit has helped too

I'm wondering when the importer seemingly stops updating or the internet is disconnected, how does XenForo handle the F5? It seems to keep track of where it is in the import process but to what degree of accuracy? Would much data be lost?

 

[Mike]

I suppose in theory it could try to import some dupes -- I'll try to check that. But if you refresh, it should pick up. The queries are almost always wrapped in a transaction, so if an error occurs, you can refresh without losing anything.

I did get some dupe errors at one point with Chrome, but not with Firefox. I haven't figured out what's going on yet.

 

[NolF]

That's excellent!! very cleaver solution I'll make sure not to use Chrome for the import once I figure out how to solve the annoying "Mysqli statement execute error : Incorrect string value: '\xAC~yaho...' for column 'account_name' at row 1" error

 

[Chromaniac]

i am facing my own unique problem. i cannot say that it is xenforo because i am facing the same problem on wordpress...

on running import... it would land on a 404. no mysql overload error. no mysql down error. a simple 404.

India Broadband Forum - Error

The requested page could not be found.

i cannot pinpoint any reason behind this mysterious problem i am facing on my server... wordpress too ends up taking me to a 404 on post publish.

i would assume that if the mysql has been overloaded, it would give out a crash/toomanyconnections related error?

 

[Rob]

That's excellent!! very cleaver solution I'll make sure not to use Chrome for the import once I figure out how to solve the annoying "Mysqli statement execute error : Incorrect string value: '\xAC~yaho...' for column 'account_name' at row 1" error

I got the same issue. Lots of people seems to have this same error at some point of the import process.... mine happens with threads and posts and it is caused by bad characters I'm sure.

 

[NolF]

I got the same issue. Lots of people seems to have this same error at some point of the import process.... mine happens with threads and posts and it is caused by bad characters I'm sure.

It's very likely. My users use lots of ACSII and Japanese characters so I wouldn't be surprised that there would be errors in the UTF-8 conversion. I hope it is something they can fix soon though, I want to play with this thing so badly

 

[steven s]

I got the same issue. Lots of people seems to have this same error at some point of the import process.... mine happens with threads and posts and it is caused by bad characters I'm sure.

Mine continue to be with PMs and Forums.