How are bigboards importing to XF?

[NolF]

By bigboards I mean lots of users and threads/posts. I have 400k users and almost 2million posts, and I cannot import smoothly.

My internet is not the most reliable of things, and compounded with having the server in the US when I'm in Australia results in many packets lost which in tern mean the upgrade process does not continue because the browser shows an error (timing out).

Is there some sort of script that can be run from the server itself (I have a dedicated server)? It should be quicker and more reliable.

I'm also wondering how do you make it seamless or reduce the downtime? Importing the table script took me over 12 hours (2Gbs), albeit the dropping pages did not help the process. You then have to add private messages, visitor messages, and the users themselves.

 

[Itworx4me]

Could happen....lol

 

[Elizabeth]

Question? Is your dedicated server a managed server? If so, that will help. Also check this thread: http://xenforo.com/community/thread...-vbulletin-forum-with-2-8-million-posts.7003/

 

[Walter]

I've done many vb upgrades, nearly half of them with the server on another continent and never had problems. But if you face packet loss your provider has a serious problem (either your ISP or your host).

And how does your server do during the import? Where is the bottleneck? CPU? Disks?

 

[NolF]

The dropped packets (mostly affect page requests) and it's due to some sort of incompatibility of the modem/router with the ISP. I'm in the process of getting a much better one which should address those concerns.

Regarding the process itself, I don't think there is a bottle neck :S Not sure how I would know. Server load is low <1, I have a Q6600 and the server has to SAS 36Gb in Raid1

 

[Elizabeth]

The dropped packets (mostly affect page requests) and it's due to some sort of incompatibility of the modem/router with the ISP. I'm in the process of getting a much better one which should address those concerns.

Regarding the process itself, I don't think there is a bottle neck :S Not sure how I would know. Server load is low <1, I have a Q6600 and the server has to SAS 36Gb in Raid1

Check out WiredTree

 

[Dean]

NoIF - I'm not an expert on servers but it looks to me like you server is not working hard at all. 85% idle, not waiting on hardware...

 

[NolF]

I've switched to opera with turbo mode on for the import process, and it hasn't timed out on me yet as chrome does. Quite interesting Still, a very slow process

Check out WiredTree

I can't afford their dedicated services as the configuration I have would cost far more than the monthly budget (paying ~250/month including management), and I don't think a VPS would suffice my requirements :/

NoIF - I'm not an expert on servers but it looks to me like you server is not working hard at all. 85% idle, not waiting on hardware...

That's what is bugging me, so many resources sitting idly...

 

[Elizabeth]

I've switched to opera with turbo mode on for the import process, and it hasn't timed out on me yet as chrome does. Quite interesting Still, a very slow process


I can't afford their dedicated services as the configuration I have would cost far more than the monthly budget (paying ~250/month including management), and I don't think a VPS would suffice my requirements :/


That's what is bugging me, so many resources sitting idly...

I'm paying less than $200 for a dedicated. It would be worth contacting the sales department and asking them.

 

[Anthony Parsons]

That's what is bugging me, so many resources sitting idly...

Not really about whether it runs low or not though... more that the resources are there for the peaks, and with over 2 million posts and 400k of users... peaks would suffice often for the solution you have, otherwise you would endup constantly locking up under a little load.

 

[AlexT]

Deleted.

 

[D.O.A.]

I set up a server at home on a decent PC with linux, downloaded the old forums backup and did it all from there, reuploaded the new/old vb attachments created by xenforo and imported the new database online

 

[NolF]

One solution would be command line support for installs/upgrades. Mediawiki supports it, and so do other apps. Especially when remotely upgrading a big board, this would be a very good way to avoid the risk of losing your connection to the server in the middle of the process. Before the upgrade you could start a session through 'screen', in which case the processes would continue to run even if you log or drop out of the shell session.

Ideally that's what I would like to have. It seems to make sense to be honest

I set up a server at home on a decent PC with linux, downloaded the old forums backup and did it all from there, reuploaded the new/old vb attachments created by xenforo and imported the new database online

My home internet is only 6/1mbps downloading processing (on a laptop) and uploading 5Gbs (including attachments) it's way too slow and cumbersome and slow.

 

[D.O.A.]

Ideally that's what I would like to have. It seems to make sense to be honest
My home internet is only 6/1mbps downloading processing (on a laptop) and uploading 5Gbs (including attachments) it's way too slow and cumbersome and slow.

sorry to hear

I'm an aussie too, but bigpond cable

I'm just gonna add another sad face cause my internet bill arrived

 

[AdamD]

Hmm, I have a dual processor, quad core machine, with 12gb of ram, if you wanna borrow that do to the import?
Let me know if so.

 

[David Gardiner]

My forum is only modest, but i did have a 200mb attachment table to handle, the only way i could get it to work was to use a proper mysql client to connect to the old database, export it, put it back very close to the new database (same server) and then run the importer, that handled it ok in beta 3.

A few things still need work, for example if you try and delete a forum with alot of subforums the job timesout.

 

[NolF]

Hmm, I have a dual processor, quad core machine, with 12gb of ram, if you wanna borrow that do to the import?
Let me know if so.

Thanks for your kind generosity, since using Opera, the upgrade process seems to have become much more reliable and faster. According to the Admin panel, 100k posts/threads imported in 5 hours, though if I go to phpmyadmin, it shows 2million posts and 2Gbs as I would expect. Weird... Importing the private messages is just as wierd though. It says 0.01% imported, back looking at the back end about 50% seems to have been imported.

A few things still need work, for example if you try and delete a forum with a lot of subforums the job timesout.

Yep, that bug seems to have stayed in beta3, we can always hide them in the mean time though

Anyways hopefully this import works out well, and I can ditch vbulletin as I'm getting some serious database errors every time I open the board