7 days to upgrade Spacebattles

[Floren]

Hi Kier,

Can you share with us what constrains you ran into SpaceBattles upgrade?
This should help other owners who want to upgrade larger forums to XenForo. From the tweets, you started the upgrade on 26th and finish it on 1st.

I recently imported a small forum (1 million posts) to XenForo on an Axivo optimized server and it took me the same amount of time. In my case the issue was mostly related to the HUGE number of attachments that needed to be imported into XenForo format (over 1ooGB). To my satisfaction, the posts were imported perfectly through the web interface.

I think a special importer should be created, that can run directly through console. It will probably avoid the browser timeouts and eventual crashes that were generated out of the blue while importing the images. These are some of the missing tasks I noticed through import:

• ability to move a number of users from a deprecated group to Registered group
• ability to mass ban users, moving them from a vBulletin Banned group and feature each user as XenForo banned
• ability to delete all threads for a specific banned user (Spam Cleaner just fails)

The main concern is: will a large site accept to have the forums closed for a week? To some that means important revenue loss and might stop them from moving the site to the new software because of that. Personally, I know people who will definitely complain if the forums are down for only 2 days.

 

[Naatan]

I'd be interested to hear what the bottleneck is in such large imports. A whole week for 1 million posts seems excessive, but if the importer is running in a single process and on a single core then that number sounds slightly more realistic.

 

[Brent W]

http://www.bigsoccer.com/forum/ 28 million posts currently going from vBulletin to xenForo.

 

[Taylor J]

It was mainly due to his internet connection which he stated in the What is going on with Spacebattles thread.

Edit: http://xenforo.com/community/threads/whats-up-with-the-spacebattles-forum.30885/#post-352719

 

[DRE]

Plus watching kids.

 

[craigiri]

While I agree that 100 GB is over the top, I think sharing information on more normal forum sizes would be a good thing.

In my case, it was about 8 GIG of attachments and 1.1 million posts.

The entire situation took less than 24 hours and we had to do TWO imports - one from EE to VB and one from VB to XF. During that time, we left the old forum up in a read-only mode (I see no reason why most forums cannot do this).

So, in effect, little or no down time for those who needed information. Less than 24 hours for those who wanted to post.

 

[Slavik]

I think sharing information on more normal forum sizes would be a good thing.

Live import: 1.2 million posts, approx 2gb of attachments, the data import took less than 6 hours, however that was running in considerably less than optimal conditions.

I have since done a second import on my test bed which ended up in an import of around 2 hours.

If I were to run it locally I think I would be able to get it to sub 30 minutes using a 64gb ramdisk.

 

[Bob]

I think sharing information on more normal forum sizes would be a good thing.

Largest conversion that I've done is: vB4->XF Threads (227,265) Posts (3,751,780) Members (60,771) Attachments (2.5 GB). Start to Finish, less than 24hrs that I had the site shut down. Closed at 6pm on a Friday, was back open for posting just a little past noon on Sat.

 

[Floris]

I remember one vBulletin 4 board with over 4 million posts and a 3 gb .sql dump compressed. Which took me three days to convert, the biggest delay was rebuilding caches. I did it localhost on my mac pro with 6gb ram, MySQL wasn't really optimized.

Any vBulletin to XenForo convert I've done with up to 500,000 posts were all done within the same day, and that included a test convert first.

Almost any convert I've done isn't done within a single day. The back and forth communication to get all the data, waiting for them to review the test convert, doing the actual convert, and then tweaking things up with search/replace in posts, plugin installation, moving over subscriptions and setting up redirects and going through permissions and options, etc. It all in all takes a couple of day, be it 250k posts or 2,5 million posts. The biggest elays have simply been doing a test, sending out the communicate that they can go test, and the next morning you get an email update saying 'yeah, I went to bed | work | emergency, will check against later tonight'. Not really because the import goes slow.

The biggest trick on bigger boards is to realize you can drop indexes and have it raw import data, and re-apply them afterwards. You cut time down by 25% to 45%. Having fast hard drives, or even multiple ones and a good MySQL configuration matching the system it runs on really helps too. Having enough memory for MySQL and PHP really helps.

I've done almost all converts on my own dedicated solution, and the bigger sites I took down localhost and re-uploaded (but I have 10mbit upstream, which helps). Every time I didn't get permission to download a backup to do the convert and they wanted to do it on their dedicated box, which usually also runs other sites, or even worse, their shared host or a vps, the convert time was taking twice as long. They're not properly configured, and as soon as the server load is high admins tend to just click reboot in the control panel, ruining everything requiring you to start over again (if not them, the host).

 

[Floren]

Live import: 1.2 million posts, approx 2gb of attachments, the data import took less than 6 hours, however that was running in considerably less than optimal conditions.

I have since done a second import on my test bed which ended up in an import of around 2 hours.

If I were to run it locally I think I would be able to get it to sub 30 minutes using a 64gb ramdisk.

Ya, in my case it was the attachments that took very long. Just curious, what do you mean by optimal conditions? I still think the best way is to have a script that runs through terminal, for large forums.

It was mainly due to his internet connection which he stated in the What is going on with Spacebattles thread.

Edit: http://xenforo.com/community/threads/whats-up-with-the-spacebattles-forum.30885/#post-352719

Thanks, I did not know this. I don't understand why he transferred the database locally, as it is a lot easier to do it on the remote server... maybe was a space issue? Transferring 7GB of data through a 1Mbit connection does takes 16hours... Personally, I created a dev.domain.com sub-domain where I ran all imports to a different database name. Then I switched the live forums to the new database.

 

[p4guru]

http://www.bigsoccer.com/forum/ 28 million posts currently going from vBulletin to xenForo.

wow didn't know bigsoccer moved over, shame some of the members aren't liking the change http://www.bigsoccer.com/community/threads/total-crap.1943197/ and http://www.bigsoccer.com/community/forums/technical-problems-and-bug-reports.165/

I disagree with Bushidou's statements, Floren's questions are legitimate and if he wasn't going to post a thread to ask this, I would of done so too (maybe not as soon as Floren did but maybe alot later). I also want to learn from what Kier learnt from his experience in migrating larger forums to Xenforo and what limitations and gotcha's we need to look out for. Floris touched on one and that is dropping indexes prior, just as with rebuilding search on large forums http://xenforo.com/community/thread...ch-index-for-a-large-site-experts-only.11195/ it does speed up things tremendously.

Thanks, I did not know this. I don't understand why he transferred the database locally, as it is a lot easier to do it on the remote server... maybe was a space issue? Transferring 7GB of data through a 1Mbit connection does takes 16hours... Personally, I created a dev.domain.com sub-domain where I ran all imports to a different database name. Then I switched the live forums to the new database.

Goes back to my statement at http://xenforo.com/community/threads/update-on-xenforo-development-status.30992/page-5#post-354197, maybe it does come down to tight finances that spacebattles forum is unable to upgrade hardware and thus constrained to the older server hardware and under vB platform they were hitting their server resource limits with no where to go, Kier hoped migrating to Xenforo would give them more breathing room on the same server hardware. This assume Kier's financially involved in spacebattles though ??? not sure. IIRC, Kier had some powerful local server/computer hardware so he probably thought that would of been quicker to process ?

Xenforo Importer Daemon mode ?
Maybe for very large forums, xenforo could have a daemon importer feature,

1. you setup a separate mysql slave server which does replication only the listening to master vb database name and replicating to slave vB database name.
2. you setup xenforo destination database name to connect from mysql slave server
3. xenforo daemon mode continuously imports (ssh cli script what Floren is suggesting) reading from mysql slave server's vB database name into xenforo database on the slave
4. live vB forum would be still online for the duration of the import as all updates/writes and reads would come from mysql master server as you don't configure vB config.php for slave usage at all.
5. the live vB forum data gets replicated to mysql slave which xenforo daemon mode continuously listens and imports from into it's own xenforo database.

Of course this moves the issue elsewhere, i.e. speed and how optimised the mysql slave server is etc

No idea how it would work in practice, just thinking out aloud here

p.s. one point Floris brought up that folks may over look, speed of source and destination hard drives. Like with back up speed, import speed also depends on how fast your disks are. If you using slooow 5400 rpm 50MB/s write speed disks, you're not going to have a very fast import heh. I had one client look into 10Gbe based remote raid array server capable of transferring 600-800MB/s - sadly cost of such build too high

 

[RobParker]

wow didn't know bigsoccer moved over, shame some of the members aren't liking the change http://www.bigsoccer.com/community/threads/total-crap.1943197/ and http://www.bigsoccer.com/community/forums/technical-problems-and-bug-reports.165/

Nobody likes change. When we moved from phpnuke to vbulletin we got loads of complaints about the change, even though it was clearly for the better. Our move to XF has been largely greeted by a lot of positive comments with a few people complaining about the lack of multi-quote, etc.

 

[Paul B]

It was mainly due to his internet connection which he stated in the What is going on with Spacebattles thread.

There was a bit more to it than that.

There were also some server related issues which impacted quite heavily on the process.

It's not really for me to say though so I'll leave it up to Kier to respond, if he chooses to.

 

[Slavik]

Ya, in my case it was the attachments that took very long. Just curious, what do you mean by optimal conditions? I still think the best way is to have a script that runs through terminal, for large forums.

We were also upgrading server, so at the time of doing the import were using a dual core with 4gb of ram. Our main server has 4x the resources available to it but was unavailable.

Also due to our site requirements we had to use customised importers which meant CLI wasn't an option at the time so everything had to be done via web interface.

 

[Nudaii]

I converted a 500k post 39k thread and 10k member forum in about 2 hours on local host, that however included converting about 2000 album pictures using the user album mod.

The uploading took about 4 hours on SLOW aol internet

 

[Floren]

wow didn't know bigsoccer moved over, shame some of the members aren't liking the change http://www.bigsoccer.com/community/threads/total-crap.1943197/ and http://www.bigsoccer.com/community/forums/technical-problems-and-bug-reports.165/

I think the users mostly complain about the small font, which could be easily fixed.

Our move to XF has been largely greeted by a lot of positive comments with a few people complaining about the lack of multi-quote, etc.

Same here, we will always get few complainers, but the end result is very positive. For example on the site I did the import, users revolted against the Facebook integration. Once I took this down, everyone was very happy. Another thing users complained was related to the speed of loaded attachments. The imported site is very heavy on images, users tend to upload 10-15 images per post (12MB each) that are automatically resized to 1280x1024px. So they complain that images do not load instantly into Image Viewer. I checked that and they are right. Technically we are talking an average of 2MB of data loaded (250KB/resized image), it takes about 30seconds to load.

 

[Floren]

As I mentioned earlier, I created a function that allows me to verify the version used on Spacebattles.
The match is:

bba17b4ab4264ea1e318029d157fb863

which translates to 1.1.3 stable version. No 1.2.0 for now, I was really hoping to see that installed on Kier's site.

 

[Robbo]

Nice script. Instead I used a URL to find what version they are using - pretty technical stuff I know. I don't know why though since it means nothing? Much like most the things you post. You are one of the worst speculators I have seen in a long time. Just give it a break, please.

 

[Floren]

I don't know why though since it means nothing? Much like most the things you post. You are one of the worst speculators I have seen in a long time. Just give it a break, please.

Here we go again... Why do you like to call names, you like to create a tense atmosphere for nothing? Some people are interested to get updates, others are not. You don't see me going after people's posts and attack them with words, while telling them what to do. Don't you think this is really immature and not productive at all? Or you think that "please" word you added at the end changes anything? These posts will probably get deleted, making the mods shrug again. Plus, you just made me waste 5min of my time for nothing.

 

[Robbo]

Here we go again... Why do you like to call names, you like to create a tense atmosphere for nothing? Some people are interested to get updates, others are not. You don't see me going after people's posts and attack them with words, while telling them what to do. Don't you think this is really immature and not productive at all? Or you think that "please" word you added at the changes anything? These posts will probably get deleted, making the mods shrug again. Plus, you just made me waste 5min of my time for nothing.

What the hell was that? I never called you anything but a bad speculator? Was it that offensive? I'm sorry you have no backbone.
By all means continue to try and act like the good guy when you are doing what was said not to do (ask for updates and roadmaps), just indirectly to get away with it.

This guy.