XF 2.0 _output vs _data. Why?

Xon

Well-known member
Am I the only one confused at why there is a distinction between these?

These look to be functional equivalent, with the major differences I can determine are;
  • _data has less files so allows uploading add-ons faster.
    • But uploading a zip and then decompressing the zip is even faster.
  • _output has a _metadata.json which contains per-file hashes, which looks to be used to detect content changes.
    • This avoid parsing & round-tripping costs, and allows templates to be edited as HTML files.
    • Could be replaced with similar metadata.json hash but for just parts of the json for _data
  • When installing an add-on, the web installer doesn't event prompt to import _output. This makes starting development of an existing add-on on a new setup annoying as you need to use the right (inconsistent) command-line incantations.
    • This goes a long way to leave a confusing mess on what is the canonical copy between _output/DB/_data instances.
For the most part, XF2 is a better development experience than XF1, but distinction between _output & _data is decidedly not pleasant and makes collaborating on add-ons harder than it needs to be.
 
Some documentation from the core developers about recommended workflows would help here.
 
  • Like
Reactions: Xon
I'm not sure why _data can't be updated at the same time _output is when in dev mode. That by itself would help.
 
or the most part, XF2 is a better development experience than XF1, but distinction between _output & _data is decidedly not pleasant and makes collaborating on add-ons harder than it needs to be.
I'm mostly just going to focus on this as I frankly disagree.

When collaborating on a project with other developers, you don't need the _data XML files. These are purely for release distribution. If you're working off a version control system, your repo likely won't host them (I'd recommend having a .gitignore file to exclude them specifically).

Any one collaborating on the project will clone the repo, copy or symlink the files to the appropriate location for development and install the add-on with the CLI command which will automatically use the dev _output files php cmd.php xf:addon-install <AddOnId>.

If the add-on is already installed, then it's just a case of importing the dev _output files php cmd.php xf-dev:import --addon <AddOnId>.

For all intents and purposes, just forget _data even exists, exclude it from your VCS repos and remember to always install via the CLI. Seems simple enough to me, IMO.
 
Just to follow up on this @Chris D - what about the scenario where we want to automate deployment from our source code repository to staging / production servers which aren't operating in development mode? (ie we don't use the release zip file for deployments).

We're going to need to include the _data directory in that case, right?

The problem is that we will need to be careful about which files are actually deployed.

My understanding is that the _output directory is basically ignored on a server not in development mode (eg staging/production), so that's not really an issue.

The issue is having the _data directory present on a dev server - which is where the potential for confusion/problems will arise.

My deployment tool can ignore files or directories when deploying - but that only helps on staging/production servers which don't matter anyway because they'll ignore the _output directory.

The real issue is when sharing files between devs or running on test machines and so on - as you mentioned, we really want to ignore the _data directory to avoid issues, but you can't easily do that when doing git pulls and such.

The only way around this is to start using release branches in my repos where the _data directory is contained in the release branch but not the _output directory, and conversely the _output directory is in the dev branch but not the _data directory.

Am I missing an easier solution here?
 
  • Like
Reactions: Xon
The issue is having the _data directory present on a dev server - which is where the potential for confusion/problems will arise.
If you're in development mode and you use the CLI tools, you're only pulling from _output -- _data is ignored. (The only time it wouldn't be is via the control panel, though we display a warning in this scenario.) We generally have both present and it hasn't caused any issues that I can recall.

Just to follow up on this @Chris D - what about the scenario where we want to automate deployment from our source code repository to staging / production servers which aren't operating in development mode? (ie we don't use the release zip file for deployments).
I guess it really depends on what you define as "deploy". If you have a build process for this, then it would really just be a matter of running a release build process on a dev server and then deploying the built package. This wouldn't really be an uncommon practice with front-end heavy frameworks where you need to do all sorts of transpiling and asset management, so it would be fairly equivalent. An alternative is to just do an export of add-on data if you don't want to do a full build. As I mentioned, having both directories is ok.

The final alternative, if you don't want to do any full build stuff and you're just doing something like a git pull, is to setup dev mode in CLI only in your config.php file. You'd have to do custom work to minify JS or other actions from your build process though which may make this more difficult.
 
I guess it really depends on what you define as "deploy"

I use DeployHQ to simply push the files from the Git repository to the server via SSH ... for XF1 I was then manually importing the addon XML file while for XF2 I was hoping it would be a simple matter of push the files, then run an XF CLI command to "import" or "upgrade" the addon in XF.

I'm trying to work out a more streamlined workflow which allows us to keep the _data and _output directories separated so that we don't run into issues in the dev environment.

I'll try running parallel develop and release branches to see if I can find something that works and isn't too cumbersome.
 
I use DeployHQ to simply push the files from the Git repository to the server via SSH ... for XF1 I was then manually importing the addon XML file while for XF2 I was hoping it would be a simple matter of push the files, then run an XF CLI command to "import" or "upgrade" the addon in XF.
It basically is though -- that's essentially what my last option was. You setup dev mode via the CLI only and do the import/upgrade command using that.

If you have other build processes though, you're skipping all of that (JS minification along with the hash generation you mentioned elsewhere).
 
It basically is though -- that's essentially what my last option was. You setup dev mode via the CLI only and do the import/upgrade command using that.

If you have other build processes though, you're skipping all of that (JS minification along with the hash generation you mentioned elsewhere).

Well then it kind of isn't simple then is it :p ... you end up losing a lot of power from the platform by taking the easy path.

I think the build process in XF has merit and can see value in that ... indeed - I already use custom build steps to generate a no-dev and optimised version of the addon's vendor folders for my composer-based addons. I don't think I'd want to try and bypass that.

I'm curious as to how you deploy official addon updates to xenforo.com ... are you only ever deploying from zip builds? Or are you bypassing that with some kind of custom build/deployment process?
 
We deploy directly from Git using git pull and we have a custom build script on the server which re-minifies any JS files. The script can do a full dev import from _output and runs xf-dev:rebuild-caches.
 
  • Like
Reactions: Sim
Most of it isn't really relevant. The meat of it is essentially
Code:
git pull
php cmd.php xf-dev:import
php cmd.php xf-dev:rebuild-caches
 
Top Bottom