XF 2.4 Improvements to database schema management

Back when we talked about what's new under the hood in XenForo 2.4, we said we'd be focusing on developer tools and features that have become commonplace in other frameworks, such as Laravel. Today's HYS is a big one in that regard: database schema migrations.

If you've used migrations in pretty much any other modern framework this will all feel very familiar. If you haven't, don't worry, the concept is simple.

What is a migration?​

A migration is a small file that describes one change to the database. Adding a table, adding a column, changing an index, converting some data, that sort of thing. Each migration has an up() method that makes the change and, optionally, a down() method that reverses it.

Migrations live in a _migrations directory in your add-on, they're named with a timestamp so they always run in the order they were written, and XenForo records each one as it runs. That means XenForo always knows exactly which changes a forum already has and which ones it still needs.

Making a migration​

There's a new make command for this:

Code:
php cmd.php xf-make:migration --addon=Demo/Portal add_view_count

Which gives you a new file, something like src/addons/Demo/Portal/_migrations/2026_10_08_141500_add_view_count.php. Fill in the up() and down() methods using the schema manager you're already used to:

PHP:
<?php

declare(strict_types=1);

use XF\Db\Migration\AbstractMigration;
use XF\Db\Schema\Alter;

return new class extends AbstractMigration
{
    public function up(): void
    {
        $this->alterTable('xf_demo_portal_item', function (Alter $table): void
        {
            $table->addColumn('view_count', 'int')->setDefault(0);
        });
    }

    public function down(): void
    {
        $this->alterTable('xf_demo_portal_item', function (Alter $table): void
        {
            $table->dropColumns('view_count');
        });
    }
};

And that's it. No upgrade step methods named after version IDs, no remembering which step number you're up to and, as with everything using the schema manager and query builder, it works whether you're running MySQL or SQLite.

Running migrations​

For admins, migrations simply run as part of installing or upgrading an add-on, exactly where upgrade steps run today, whether that's in the admin control panel, the CLI or a batch install.

For developers there are a few CLI commands to make life easier while you're working:
  • xf-migrate:run runs any pending migrations
  • xf-migrate:status shows which migrations have run and which are pending
  • xf-migrate:rollback rolls back the last batch, for when you need another go at a migration you're still writing
Migrations also aren't tied to your add-on's version number. Each one is tracked individually, so you can add a migration, run it, tweak it, roll it back and run it again without bumping your version every time. If an add-on has a pending migration, XenForo will offer to upgrade it even if the version hasn't changed.

Bigger changes​

Some changes need to touch a lot of data, say backfilling that new column for every row. Rather than trying to do it all in one go and hoping the request doesn't time out, a migration can implement a batch() method and work through the data a chunk at a time:

PHP:
public function batch(MigrationBatch $batch): bool
{
    $ids = \XF::query('xf_demo_portal_item')
        ->select('item_id')
        ->where('item_id', '>', $batch->position)
        ->orderBy('item_id')
        ->limit(500)
        ->fetchAllColumn();

    if (!$ids)
    {
        return true;
    }

    // ... update this batch of rows ...

    $batch->advance(end($ids));

    return false;
}

Progress is saved after every batch, so if an upgrade is interrupted it just picks up where it left off. If you need something to run outside of the migration itself, migrations can also queue jobs, either to complete before the add-on is reactivated (insertImmediateUpgradeJob()) or to run in the background afterwards (insertPostUpgradeJob()).

What does this mean for admins?​

Quite importantly, not much :) Installing and upgrading works just as it does now. The main thing you might notice is that larger upgrades are a bit more forgiving, as they work through in smaller steps and resume if something interrupts them.

What does this mean for developers?​

First of all, nothing you have today stops working. Your Setup class, its install steps and upgrade steps all work exactly as before, and you can start using migrations whenever you're ready.

A few things worth knowing when you do:
  • On a fresh install, your Setup install steps run first and then your migrations run on top. So the simplest approach is to leave your install code as it is and make any future changes as migrations. A brand new add-on doesn't even need install steps at all, it can just be migrations.
  • If your install code already builds your latest schema, you can return true from installIncludesMigrations() in your Setup class and your migrations will be marked as done on install rather than being run.
  • Uninstalling still goes through Setup::uninstall(). Migrations are not rolled back when an add-on is uninstalled.
  • Once your add-on has migrations, xf-addon:build-release won't let you add new upgrade steps for the version you're releasing. Upgrade steps always run before migrations, so mixing the two could mean an upgrade step runs before the change it relies on.

Supporting XenForo 2.3 and 2.4 at the same time​

XenForo 2.3 doesn't know what a migration is so it will simply ignore your _migrations directory. If a single release of your add-on supports both 2.3 and 2.4, any database changes still need to be upgrade steps so that forums on 2.3 get them too. Upgrade steps continue to work in 2.4, so this is completely fine.

So our recommendation is:
  1. While your add-on supports XenForo 2.3, keep using upgrade steps.
  2. From the first release that requires XenForo 2.4 (2040010 in the require section of your addon.json), switch to migrations.
There's no need to convert any of your existing upgrade steps. They stay where they are and keep doing their job for anyone upgrading from an older version.

And XenForo itself?​

We've moved over too. All of the database changes in XenForo 2.4 are written as migrations and our install data is now generated from them, which, as a nice side effect, means a fresh install and an upgraded forum always end up with exactly the same database.

1791547843482.webp


As ever, let us know if you have any questions below. We're looking forward to seeing what you do with it!
 
Cool stuff. Does this mean we can downgrade now? Like go from V x+1 to x?
I mean technically that's slightly easier with (many) caveats but not really what it's designed for, no. Bringing a migration "down" is more of a developer environment thing than anything else.
 
Moving away from magic ids in function name is definitely appreciated.

This process also works much nicer for branch-based development as the migration is by-date and not by an ID which isn't yet fixed.

@Chris D does this mean XF tracks which add-on migration has been evaluated (and if any errors occurred) in a table somewhere?
 
Last edited:
Thanks! Yeah, the branch-based development point is a big one for us too of course. Even though we’ve had named upgrade steps during development for a while which has helped, it’s still a source of conflicts quite often.

Yes, there's a new xf_migration table. Each migration that has run gets a row with the add-on ID, the migration's file name and the batch it ran in (all migrations applied in a single run share a batch number, which is what xf-migrate:rollback uses). Migrations are tracked by name rather than by version, so working out what's pending is just "which files in _migrations don't have a row yet".

Errors aren't stored in the table as such. A migration is only recorded once it completes successfully, so if one throws an error, the error is shown just like any upgrade error is today and the migration stays pending. So you just fix the issue, run the upgrade again and it picks up from that migration.

Worth ensuring the migrations are idempotent in case a failure requires a rerun that leaves things in an inconsistent state (although schema manager should handle this for the most part).

For migrations that use batch() there's also an xf_migration_progress table, which saves the current position after each batch. Each batch and its progress update are committed together, so if a batch fails it's rolled back and the next attempt resumes from the last batch that succeeded rather than starting again.

You can see where any add-on is up to with:

Code:
php cmd.php xf-migrate:status --addon=Vendor/AddOn
 
@Chris D is there a replacement for postUpgrade<VersionId> for migrations? Sometimes an expensive rebuild might need to be enqueued after a migration is completed 🤔
 
There doesn't need to be a replacement for something we're not replacing :)

Your Setup class, its install steps and upgrade steps all work exactly as before, and you can start using migrations whenever you're ready.

Just carry on using Setup::postUpgrade() as before :)
 
It would still be nice to have it so we don’t need to keep adding branches to it, like a postMigration function that would get called on all migrations during postUpgrade.

I realise we could implement that ourselves but native would be nice 😄
 
Yes, there's a new xf_migration table. Each migration that has run gets a row with the add-on ID, the migration's file name and the batch it ran in (all migrations applied in a single run share a batch number, which is what xf-migrate:rollback uses). Migrations are tracked by name rather than by version, so working out what's pending is just "which files in _migrations don't have a row yet".

Errors aren't stored in the table as such. A migration is only recorded once it completes successfully, so if one throws an error, the error is shown just like any upgrade error is today and the migration stays pending. So you just fix the issue, run the upgrade again and it picks up from that migration.
I think it is important that a failed migration can block other future migrations, I assume it just iterates in lexical order of the files? So if a migration fails the installer/upgrader just aborts and gives an error?
 
Yeah, they run in order based on the name. If one fails, the upgrade stops there with the error, the migration isn't recorded, and nothing after it runs until it succeeds.

The only exception is a migration that opts out via canRun(). They are skipped and stay pending rather than blockig.
 
Oh I like this, I have been doing lots of work with Laravel recently and this is very familiar.
 
No a developer may be able to rollback their migrations but there isn’t a “downgrade” feature. That’s not a thing that we want nor anyone should.
 
So what’s the point of migrations, then? If you can't roll back? How is this mechanism any better than the commands I currently run to perform a rollback? Just to reduce the amount of code in Setup? Although it's more about reducing merge conflicts when people write the same update function.
 
To be clear, migrations can be rolled back. You can see that clearly for yourself if you read the post.

What I’m saying is you can’t “downgrade” an add-on. Like a user can’t install version 2.0 and expect to downgrade to 1.9. That’s the question you asked.
 
Yes give me more Rails/Laravel inspired dev features it feeds my soul, especially since I do nothing but work in Rails all day.

Should probably setup a suggestion thread for this but so I don't forget it would be nice to have some little oddities from rails/laravel on Finders.
Code:
->findFirst()
->findLast()
->last(10)

Would all be nice + some others I'll add to that thread.
 
Back
Top Bottom