Large Forum - VB4 to Xen 2.. Questions

[MentaL]

Hi

I've got a moderately large forum I'm wanting to convert to Xen 2 sometime in the near future but my main issue is how stable is the conversion? Example, several million threads / posts need to switch over and I'm not wanting to run it via a standard web interface but maybe CLI so there are no time out issues. Is this something Xenforo is prepaired for or do I need to buy that Bigboard migration software?

Cheers.

 

[RobParker]

XF will handle it fine via the CLI. You'll obviously want to do a test conversion before going live but it should all go smoothly.

 

[Chris D]

Your best bet at this point is to use the built in importer and use the CLI to do it, and as Rob says above, definitely do some testing with it. If you hit any issues with the import, we should be able to help.

We don't do imports ourselves and as far as I know the big board importer isn't currently XF2 compatible (yet). but the built in CLI one should be up to the job

 

[MentaL]

Cheers guys.

 

[MentaL]

Test import been running two days, very very slow. Even via CL it's just slow, maybe I need to reconfigure the database server a bit as it keeps dropping and I have to restart.

 

[RobParker]

I'd have expected a couple of hours at most!

 

[MentaL]

- Step 16 of 26: Posts 33:59:05 [1,610,377] 17.01%


This is my life now.

 

[Chris D]

I’d definitely expect it to be quicker than that, even if it is over 9 million posts.

 

[MentaL]

I tweaked the mysql config, seems to go a lot faster now.

 

[Ozzy47]

What percentage is it at now?

 

[MentaL]

- Step 16 of 26: Posts 36:42:30 [2,456,587] 25.34%

 

[ZAZmaster]

I tweaked the mysql config, seems to go a lot faster now.

Prompt please what changes made in mysql

 

[fotografik]

- Step 16 of 26: Posts 36:42:30 [2,456,587] 25.34%

Hi,

We just recently (a week ago) moved our VB4 forums to XF2 and we experienced the exact same issue with the built-in importers during our testing phase. In our case, we had 9M posts and 1.5M threads to migrate so the CLI built-in importer was not going to cut it.

Ultimately, what worked for us was a 2-step migration process using the Big Board Importer that was developed for XF1.x. We did the initial import into XF1.5, did some configuration on the Administration side, and then ran the upgrade to XF2.

We ran through this process several times just to make sure that all steps were taken, and the full migration (old forum shutdown, database backup, import and upgrade, then styling and final configuration) took about 8 hours in total.

Very happy that we made the move, our members love the new features that were not available in VB4 and the site overall runs a lot better/smoother on a smaller server. We are still working out some kinks for example our search index rebuild is taking a very long time, and we have a ton of old indexed and hard-coded links that need to be redirected to the new XF2 URL format.

Hit me up if you need more specific information on how we did the migration.

 

[ZAZmaster]

fotografik, your experience is very useful, but not everyone has money on the Big Board Importer

 

[fotografik]

fotografik, your experience is very useful, but not everyone has money on the Big Board Importer

No worries, just sharing my experience. Each board owner will have to decide the best way forward for themselves. In our case, the built-in importers did not present a workable migration path.

 

[RobParker]

In our case, we had 9M posts and 1.5M threads to migrate so the CLI built-in importer was not going to cut it.

Did you try? I'm a bit surprised as the web-based importer for us handled 5M posts fine (albeit a bit slow). I imagine that the CLI importer would have been fine.

 

[fotografik]

Did you try? I'm a bit surprised as the web-based importer for us handled 5M posts fine (albeit a bit slow). I imagine that the CLI importer would have been fine.

Yes, we did try as the single-step import from VB4 to XF2 would have been preferable - tried at version 2.01 then at version 2.02, but did not try at version 2.04 though. However, our circumstances are likely different as we are running on AWS EC2 and RDS instances for the web and database respectively.

We tried different combinations of EC2 and RDS instances (more memory, more network capacity, SSD storage) but the results were very similar across different types of instances – posts would literally be importing at a rate of 1-2 posts per minute.

We even tried using a single large server instance and moved the database into the same server, but results were not that much better.

The AWS environment is likely very much different from other hosting providers so there likely is a higher "cost" in terms of latency between the server and database.

In saying that, however, the Big Board Importer queried the exact same database instance(s) and it could extract (and then insert) all the data and from the database monitoring stats, the usage (CPU, Write IOPs, Read IOPs, networking throughout) was well within the limits of the database instance.

As always, YMMV.

 

[Slavik]

Did you try? I'm a bit surprised as the web-based importer for us handled 5M posts fine (albeit a bit slow). I imagine that the CLI importer would have been fine.

Its worth mentioning, the CLI importer while it may be a little quicker, is still essentially running the web-based import system just without the web end attached, there isn't anything overtly "coded" to make it go faster, just easier.

 

[RobParker]

Its worth mentioning, the CLI importer while it may be a little quicker, is still essentially running the web-based import system just without the web end attached, there isn't anything overtly "coded" to make it go faster, just easier.

Maybe it’s the default settings being different but it’s certainly considerably faster for us