Build-release independant from database

[QuackieMackie]

This suggestion is being made to re ignite an old suggestion that had a lack of interest. https://xenforo.com/community/threa...without-needing-a-full-xf-environment.187253/

---

As the title suggests, I’d like to see the php cmd.php xf-addon:build-release Vendor/Addon command function independently of the XenForo database.

At the moment, this isn’t possible because xf-addon:build-release depends on xf-addon:export, which requires a live database connection to generate the _data directory.
This dependency makes it impossible to perform a full release build in environments where a database isn’t available.

Removing XenForo’s current database dependency when building a release would mean developers would be able to automate their workflows, for example:
- Automatically build and package add-ons in CI/CD systems (e.g., GitHub, GitLab) when changes are pushed or a release is tagged.

Proposed solution
There are a couple of approaches, I could think of:

• Keep _data automatically up to date whenever add-on files change, so xf-addon:build-release can run without accessing the database.
• Allow xf-addon:export to operate directly from add-on files and the existing _output directory, rather than requiring a live database, and generate _data from that.

 

[Taylor J]

I'd also like this.

I have been tinkering with self hosting a Gitlab instance and trying to get a full XF env also hooked up to it to do exactly this, but haven't had the time to fully even figure it out and get it working with looking for a job.

 

[QuackieMackie]

I'd also like this.

I have been tinkering with self hosting a Gitlab instance and trying to get a full XF env also hooked up to it to do exactly this, but haven't had the time to fully even figure it out and get it working with looking for a job.

Self hosting GitLab should in theory make it a lot easier to do.

Whether it’s easy though is another matter. If you do end up getting it working I’d love to know what you ended up doing.

 

[Kirby]

I don't think implementing this suggestion would be worth the effort (if possible at all):

1. Many (most?) 3rd party Add-ons seem to be developed by individual developers who primarily do this on a hobbyist basis that often do not use CI but build directly from their development instance (maybe even without using version control at all)
2. While not straightforward, it is already possible to use CI toolchains to build XenForo Add-ons, we've been doing this internally for years using PHP Censor pulling from a self-hosted GitLab instance (could also pull from GitHub, etc.).
   Using GitLab, GitHub (or basically any tool that uses docker images) runners is also possible, you just have to use a custom build image that already has a XenForo installed or first install XF before building the Add-on (takes a bit longer)
3. XenForo 2.4 will support SQLite additionally to MySQL. Using SQLite the database will just be one additional file which significantly simplifies an automated build process
4. There might be Add-on code that needs to run during the build process, either from the Add-on itself or from other Add-ons.
   How should this be done if the Add-on isn't installed and has access to its data (in the database)?

 

[QuackieMackie]

I don't think implementing this suggestion would be worth the effort (if possible at all):

I think your right. Without a lot of work, setting up any of the build process to be automated is a huge pain, hopefully with 2.4 this does get easier

I’ve been experimenting with self-hosted runners today and got to the point where the workflow can build the add-on and upload the artifact to GitHub. However, the approach has a major limitation. If, for example, I’ve pushed changes X and Y but left file Z unfinished in my dev environment, the workflow will include whatever is currently present on disk. That means the release ZIP could be incomplete or incorrect, which makes this approach unreliable for real releases.

 

[Kirby]

If, for example, I’ve pushed changes X and Y but left file Z unfinished in my dev environment, the workflow will include whatever is currently present on disk. That means the release ZIP could be incomplete or incorrect, which makes this approach unreliable for real releases.

I am not sure if I fully understand what you are talking about.

Normally a workflow IMHO would

1. Clone the commit
2. Lint
3. Install the Add-on in a pristine XenForo environment (either using a docker image that has a clean XenForo already installed or by installing a fresh XenForo first)
4. Run (unit) tests and tools like PHP CS Fixer
5. Build the release (if everything went fine so far)
6. Upload / publish artifacts

So what do you mean by "currently present on disk"?

 

[QuackieMackie]

I am not sure if I fully understand waht you are talking about.

Normally a workflow IMHO would

1. Clone the commit
2. Lint
3. Install the Add-on in a pristine XenForo environment (either using a docker image that has a clean XenForo already installed or by installing a fresh XenForo first)
4. Run (unit) tests and tools like PHP CS Fixer
5. Build the release (of everything went fine so far)
6. Upload / publish artifacts

So what do mean by "currently present on disk"?

I tried combining my dev enviroment, and the selfhosted runner so I wouldn't have to set up a seperate Xenforo site. Not the best idea, but I was being lazy.

 

[Kirby]

Ah
Yeah, I wouldn't do that as it's calling for trouble - always use a fresh instance for every build run.

 

[Taylor J]

I am not sure if I fully understand what you are talking about.

Normally a workflow IMHO would

1. Clone the commit
2. Lint
3. Install the Add-on in a pristine XenForo environment (either using a docker image that has a clean XenForo already installed or by installing a fresh XenForo first)
4. Run (unit) tests and tools like PHP CS Fixer
5. Build the release (if everything went fine so far)
6. Upload / publish artifacts

So what do you mean by "currently present on disk"?

Hmm I hadn't thought about this, for the docker way do you just have it so that you just use the same files but a new db/etc every time or always replace the xf files as well?

 

[QuackieMackie]

I did a few days of off and on again tweaking before I finally have something working how I want.
So a few things I did to set this up:

• I set up a completely clean Xenforo installation on a self hosted device.
• I set up my own github runner so I can just access the Xenforo installation without needing to upload it anywhere, or mess around with docker and such.
• I created my own actions for the workflow.

Now the results of this are Sylphian-Network/xf-addon-actions. This comes with 3 actions prepare-install, cleanup, and build-release. More will be added when and if I need them.

This let's me create a workflow like this:

name: Build Sylphian/Backup Add-on

on:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  build:
    runs-on: self-hosted

    steps:
      - name: Checkout workflow repo
        uses: actions/checkout@v4

      - name: Prepare & Install addons
        uses: Sylphian-Network/xf-addon-actions/actions/prepare-install@v1
        with:
          main_addon: Sylphian/Backup:https://github.com/Sylphian-Network/Sylphian-Backup.git
          dependencies: Sylphian/Library:https://github.com/Sylphian-Network/Sylphian-Library.git
          base_dir: /var/www/github.runner
          gh_token: ${{ secrets.GH_TOKEN }}

      - name: Build release
        id: build
        uses: Sylphian-Network/xf-addon-actions/actions/build-release@v1
        with:
          main_addon: Sylphian/Backup
          base_dir: /var/www/github.runner

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: Sylphian-Backup-${{ steps.build.outputs.version }}
          path: ${{ steps.build.outputs.release_path }}
          if-no-files-found: error
          retention-days: 14

      - name: Cleanup
        if: always()
        uses: Sylphian-Network/xf-addon-actions/actions/cleanup@v1
        with:
          main_addon: Sylphian/Backup
          dependencies: Sylphian/Library
          base_dir: /var/www/github.runner

Which produces an artifact containing the zzip file produced by the build-release command

This was my first time experimenting with custom actions, and self hosted runners so theres probably a bunch of mistakes.
If anyone ends up attempting something similar and improving what i've done at all I'd love to hear notes.​

 

[Kirby]

This kinda works, but only as long as an Add-on doesn't break the instance, aborts during install or uninstall etc.
If it does, your build environment is broken and requires manual fixing.
IMHO automated build runs must never persistenly break the build environment, no matter what happens - they therefore should be ephemeral.

@Taylor J
There are multiple options:

1. Install a fresh XenForo instance from scratch for every build by pulling a XenForo installation ZIP from an external source
   This is the simplest approach and doesn't need a custom image, you could even pull the ZIP directly from api.xenforo.com (tough I don't know how happy XenForo would be if you do this frequently ...)
2. Use a custom image that includes the install ZIP
   Pretty much the same as 1), just doesn't use an external source to get the ZIP
3. Use a custom image that has XenForo unpacked and install from there
   Very similar to 2) but avoids unpacking for every build run
4. Use a custom image that has a fully installed XenForo + DB dump
   Upon build run this just requires importing the dump
   This is faster than all previous options, but still requires a bit of time to import the dump
5. Use a custom image that has a fully installed XenForo and MySQL / MariaDB
   This is a bit against Docker principles ("One container per process"), but should work fine and is the fastest option as everything is already installed and just needs to be started

Our build processes (which does not use Github actions so far) is basically build around option 1).

With XenForo 2.4 this should become even easier as no database server is required.