XF 1.5 Upgrading large forum from XF1.5 to XF2

[Apt Irrelevance]

I'm starting preliminary research on what it will take to upgrade our forum to the latest XF2 release. Our last major upgrade was in December 2017 and we decided to hold off on XF2 simply because some of the crucial add-ons we use weren't yet supported. (The irony isn't lost on me that we upgraded to XF 1.5 right as 2.0 was coming out and now we're finally looking to upgrade to XF2 so close to the release of XF3.)

I understand XF2 is a massive code rewrite, but does it significantly alter the database as well? I'm curious about how much the size of the database matters in how long the upgrade will take. We are an old forum with close to 30M posts, 270k users, 475k threads, and currently 327 nodes. When we initially imported our database into XF 1.1 from IGN boards the process took several days, mostly because of the DB size but also I believe the dev had to make some adjustments to the custom script because some of the posts weren't importing.

Some of the major add-ons we use that we're looking to upgrade are:

• User Essentials
• Conversation Essentials
• Conversation Improvements
• Multi Prefix
• [Extra License] Usernotes
• [SSD] Custom Markup For Users

So with that in mind, should I expect a significantly longer downtime than a typical upgrade of XF1 to XF2? In our community, forum migrations and upgrades are notorious for taking way way longer than initially promised, so this time around I'm hoping to get a better understanding of what to expect before we finally pull the trigger on the upgrade so we can better manage expectations.

Any advice in regards to these particular add-ons would be appreciated as well. Most of them it appears have straightforward upgrade processes, but I'm not sure whether the last two have XF2 equivalents or can be easily upgraded.

Thanks

 

[Suzanne O]

I suggest you get one of the staff to upgrade your forum this time.
Just so they can show you how it’s done.
you might want to read this and use this if you’re wanting to do this.

Solutions

Tools and services to help you serve your community.

xenforo.com

 

[goyo]

It's so hard - obviously - to give a definitive answer as it depends on a myriad of factors - your server's speed and PHP/Memory settings, database inconsistencies, number of attachments, poorly written add-ons, etc.

A few years back, I migrated a forum with 600k+ users (but only 5M posts) using

php cmd.php fx:upgrade

In that case, it took me only a few hours without the add-ons - and a few minutes rebuilding Elasticsearch with CLI.

 

[Apt Irrelevance]

I'm not sure of our current server's capabilities but we did upgrade to a beefier one with significantly more CPU threads and memory a year or two ago.

What sort of things would cause database inconsistencies? The import issues we originally had may have been due to data from the oldest iteration of our forums back prior to 2001. Data from the original software, UBB, was largely corrupted prior to that migration from what I understand with lots of user data and posts being lost.

We don't actually have very many attachments, as they are turned on only for moderators and admins. We have about 1900 total and they're all images.

This is a full list of our installed add-ons. I want to say we haven't really had anything else installed other than older versions of a few of these:
Add-on Install & Upgrade 1.4.3
Bb Codes & Buttons Manager 3.3.5
Citas de XenFacil 9.0
Conversation Essentials 1.12.20
Conversation Improvements 1.3.14
Deadlock Avoidance Hack 1.0.20
Email Queuing Enhancements 1.1.1
Forum Moderators 3.1
ForumRunner for XenForo 1.1.4
IGN Snowboards CLI Importer 1.0
Multi Prefix 1.8.1
Nodes As Tabs 1.5.1
Optimized List Queries 1.3.5
Prefix Forum Listing 1.2.5
Quick Reply Preview 1.4.4
s9e Media Pack 20180204
Tapatalk 3.2.4
Toggle ME 3.1.4
User Essentials 2.7.9
User Merge Search Update 1.1.0
User Promotion on update 1.1.2
XenForo Enhanced Search 1.1.8
[bd] Tag Me 3.2.0
[bd] Widget Framework 2.6.3
[Extra License] Usernotes 1.2.4
[SSD] Custom Markup For Users 1.0.1

 

[MySiteGuy]

Your UBB database almost certainly used MyISAM table types, which are not crash safe. If they weren't checked and properly repaired/restored, corruption on a large forum was bound to happen. XF primarily uses Innodb table types, which are crash safe in almost every case. There might be logical inconsistencies over time due to XF or add-on bug.

No one can say exactly how long it will take with any certainty before actual test imports are run due to too many variables. I've migrated hundreds of forums from hundreds of posts to massive boards like yours - my experience the same forum on the same hardware, the speed can vary by an order of magnitude with simply better database tuning. You won't know until you do a test import.

Some of those add-ons don't store additional data, so there isn't extra downtime for them. Others don't have an XF 2.x equivalent, or are built in features now that need to be setup manually (nodes as tabs) so downtime can be ruled out for them. Still others, like Tapatalk... it's a good time to consider getting rid of them due to 2's improved PWA features.

For a board that big, consider staging a full import for testing, doing a few incremental/delta imports during testing, then a final incremental/delta import when going live. This should minimize downtime to an hour or less, versus a day or more. I've used my incremental XF 1.5 to XF 2.2 importer to do this many times - not only does that allow a testing site as long as needed, it lets you turn that into the live site at the end with very little down time.

Also, to answer another one of your questions - the XF 1.5 and XF 2.2 database schemas are very close. Off the top of my head, I'd say 95% identical.

 

[Apt Irrelevance]

Thank you for that info. The UBB data being MyISAM would certainly make sense considering how little of that data ended up surviving. Sounds like we need to look into database tuning before starting. I'm sure we've already done a little since moving to Xenforo, but it may have been years at this point since we've given it a good look. One of the joys of running a site with volunteers is finding out things that should be routine maintenance sometimes get neglected for months or years before someone else notices. We've had good server admins over the years, but it's anyone's guess how much upkeep has been done along the way.

Staging a test import and following up with delta imports sounds like the way to go for us. It's likely going to take a while to get all our templates and add-ons figured out anyway and we'll have a much better idea of how much downtime to expect for the final import.

 

[zappaDPJ]

I would delete ForumRunner's files or at least do some research on it. As far as I'm aware it's no longer supported and may well contain an SQL injection vulnerability.

I also agree with @MySiteGuy, it's now worth considering removing Tapatalk. I'm also sure @MySiteGuy could handle your upgrade for you should you require a professional service.

 

[Apt Irrelevance]

I would delete ForumRunner's files or at least do some research on it. As far as I'm aware it's no longer supported and may well contain an SQL injection vulnerability.

Agreed on ForumRunner, it appears to no longer exist. I know we still have users who use TapaTalk daily though.

 

[MySiteGuy]

LOL, was just about to post "does ForumRunner exist anymore?"

 

[TPerry]

LOL, was just about to post "does ForumRunner exist anymore?"

The pity is... when it was actually ForumRunner, it was a better product than TapaTalk. I hated when it got sold.

 

[TheLaw]

Some of these issues are not so simple such as if you have any type of customized forum theme as they are all but guaranteed not to work properly. We have a customized, two level header which helps with global navigation on top and local navigation on the bottom. Then there is your home page layout if you have a customized one, like we do. This is the first issue I'll be tackling with our major upgrade this month.

Next is the plugins issue. The most major ones are still supported but perhaps not several of the smaller ones. Some have been superceded by other advances of the software but others will not be properly consolidated into the upgrade. This is where it becomes a major chore as we have had to identify the most important functionality we have and need and what might be lost in an upgrade.

I don't see any shortcut to this except to do a site upgrade on a test site. Document everything you're doing. See what works and what doesn't. Complete the dummy upgrade. Then have a plan to upgrade the site expeditiously based on your documentation during optimal downtime, e.g., nights and weekends. If you want to use a repository and move that over that's probably fine but I prefer not to work that way since I'm not always sure what files may get modified during an installation and such can create havoc.

This is my approach but there are probably others who can give better advice on how they upgrade.