What's new for developers in XenForo 2 (part 2) (December 30th) (@Mike)
A couple of weeks ago, we went into some detail about what's new for developers in XenForo 2. (You can read that
here). We promised we would be back for more and so here it is:
Add-ons
In XF1 there are very few conventions and little structure in terms of the naming of add-ons, where the files are stored and what those files are named. We are changing that in XF2 in quite a significant way.
First of all, there is now a specific directory for all add-ons. Each add-on will have its own directory inside this parent add-ons directory and each directory will be identified by the add-on ID of the add-on. For example, files for XF Resource Manager and Media Gallery will be in the
addons/XFRM and
addons/XFMG directories respectively where
XFRM and
XFMG are the new add-on IDs for these add-ons.
On a similar note, a common pattern among existing add-ons is for an add-on vendor to put all of their add-ons within a parent directory indicated by the vendor's name. For example, if you release add-ons under the "Acme Add-ons" brand, you may currently store your add-ons files in the
library/AcmeAddOns/AddOnName directory. Your add-on IDs may even be something like
AcmeAddOns_AddOnName. In XF2, we support add-on IDs having a vendor prefix as part of the add-on ID, and the vendor prefix is separated by a
/. So, if an add-on ID contains a slash, e.g.
AcmeAddOns/AddOnName we know the files for the add-on will be stored in
addons/AcmeAddOns/AddOnName. As we now use PHP namespaces in XF2, the rest of the add-on ID should be compliant with that format, so add-on IDs should contain only
a-z,
A-Z,
0-9 and
/ characters, must not start or end with a
/, and must not start with
0-9.
This convention is certainly less flexible, by design, but having a predictable format for such things represents a significant improvement. For example, all add-ons will now have nearly identical install instructions, manually removing add-on files will be a clearer process and it allows us to make the process of installing add-ons different, too.
With that in mind, the new process for installing or upgrading an add-on is to simply upload the contents of the
upload directory to the
add-ons directory on your server. After you have done that, you no longer have to fish out a specific XML file to trigger the install or upgrade - we already know where the required files for an add-on are.
To demonstrate, this is a good point to show you what the new add-on list looks like:
The add-on list is split into a few separate lists of add-ons.
When an add-on directory contains a version of files newer than that of the version installed, we mark it as upgradeable and push it to the top of the page.
When an add-on exists on the file system but is not installed (not in the xf_addon table of the database), we pull out some information about the add-on and mark it as installable.
Finally, you will see a list of add-ons which are installed, though you will notice that one of those is disabled.
Of course in XF1 all of the data for an add-on is kept inside an XML file. In XF2, each add-on should provide an XML file (if applicable) for the various types of add-on data (phrases, templates, etc.) and this file is stored in the add-on directory and named
data.xml. This file no longer stores information about the add-on title and version. Instead, every add-on should include a manually built
addon.json file.
Here's a somewhat complete example from XFMG 2.0:
JSON:
{
"legacy_addon_id": "XenGallery",
"title": "XenForo Media Gallery",
"description": "The XenForo Media Gallery is an add-on that allows you and your users to create galleries of images and videos in your forum, organized into admin-defined categories or user-created albums.",
"version_string": "2.0.0 Alpha",
"version_id": 902000010,
"dev": "XenForo Ltd.",
"dev_url": "https://xenforo.com/",
"faq_url": "https://xenforo.com/help/media-gallery/",
"support_url": "https://xenforo.com/community/forums/media-gallery-support.87/",
"extra_urls": {},
"require": {
"XF": [2000010, "XenForo 2.0.0+"],
"php": ["5.4.0", "PHP 5.4.0+"]
},
"icon": "icon.png"
}
For the most part, this should be fairly self-explanatory based on what you can see in the above screenshot, so we won't go through everything. Though there are a couple of interesting things here.
- We mentioned earlier that the add-on IDs for XFMG and XFRM have changed, and this add-on ID change is handled automatically during upgrade based on the
legacy_addon_id key in the JSON file. If that add-on ID exists, it will be marked as upgradeable by this add-on.
extra_urls is an array of link text: url pairs and these will render below the add-on list item (e.g. after the developer, FAQ and support links) and allow you to display links to other relevant things related to the add-on (perhaps a bug reports link, a manual or whatever you like).- The
require array is a standard way of blocking an add-on install or upgrade if the environment doesn't support the required dependencies. In this particular JSON file, we block install and upgrade if the current XF version ID is below 2000010 and we block if the current PHP version is below 5.4.0. It's also possible to require other add-ons to be installed first, and also block install if certain PHP extensions are missing or not enabled.
- The
icon path is relative to the add-on directory. In this example, our XF logo is picked up from addons/XFMG/icon.png.
Another thing missing from both the XML file and the JSON file is information about how to install/upgrade the add-on. In XF1 we specify a class and method for installing and uninstalling an add-on. In a perhaps somewhat predictable move, we have also made this somewhat more predictable in XF2
If you have any install/upgrade/uninstall operations that need to happen, you will add these to a file named
Setup.php. There is a
BaseSetup class that you can extend here which can, amongst other things, loop through install steps and upgrade versions automatically.
PHP:
public function installStep1()
{
// Do stuff here
}
Development output
Although somewhat still related to add-ons, this is significant enough to deserve its own section.
If you enable development mode in config.php, this will cause add-on related data to be output as files (usually JSON) to the file system. Files will be written out to a location in the format
addons/<addon_id>/_output/<type>/<id>.json.
For example, this is the XFMG navigation tab entry, stored in
addons/XFMG/_output/navigation/xfmg.json:
JSON:
{
"parent_navigation_id": "",
"display_order": 30,
"navigation_type_id": "basic",
"type_config": {
"link": "{{ link('media') }}",
"icon": "",
"display_condition": "$xf.visitor.canViewMedia()",
"extra_attributes": []
},
"enabled": true
}
Outputting all associated development data in this way is perfect if you manage your projects using a version control system. It also provides a way to easily import new/updated development data when working across separate databases, without having to do a full add-on install/upgrade.
As well as outputting the files themselves, we also track the metadata for all file types which at the minimum will include a hash which represents the current content of each file but can also include information such as the version ID and string of the add-on when it was last changed.
Not all development output files are output in JSON format. For example, Phrases are output as
.txt files but, perhaps more significantly, template files are output in
.html,
.less and
.css format...
Template file watcher
As mentioned above, templates are written out to the filesystem using the development output system. The location for these files will be
addons/<addon_id>/_output/<admin|email|public>/. As each template is rendered, we watch the file system to see if there are any changes by comparing the current hash of the content to the hash stored in the development output metadata. At this point, the template will be updated, imported or recompiled.
This essentially means that the ability to edit templates from the filesystem is available for add-on developers which is far more convenient than editing templates in the Admin CP.
Class extensions
In XF1, to extend certain classes, you have to go through the following process:
- Create a listener class, and a method similar to:
PHP:
public static function extendThreadController($class, array &$extend)
{
$extend[] = 'My_Extended_Thread_Controller';
}
- In the Admin CP create a new code event listener, pick an event, e.g.
load_class or load_class_controller and point it at your class and method and provide an event hint.
This doesn't particularly seem too onerous, but with a lot of classes being extended it can ultimately represent a lot of boilerplate code - and where's the fun in that?
So, in XF2, we have simplified this into a system in the Admin CP we call "Class extensions". Let's take a look:
All you need to do then is create your extended class, and that's it! It is, however, important to bear in mind that wherever possible a code event listener should still be used to listen to more specific events if there is one available.
The app object, dependency injection and service location
XF2's application config and setup actions are now configured through a general app object. (This is similar to XF1's dependencies objects, though the app object is more encompassing.)
The most significant component of this object is the dependency injection container that it exposes (and the helper methods it provides to work with it). Without getting into too much detail, the main advantage of this container is that it wraps up the dependencies for creating various objects within XenForo (or loading specific types of data) so you don't need to worry about that; you'll always get the correctly setup object. (If you're not familiar with dependency injection or containers, have a look at the first 2 parts of this blog:
http://fabien.potencier.org/what-is-dependency-injection.html)
As an example, if you request the router for generating/analyzing public routes, the container will load the actual list of routes, setup the pre-processors, add the route filters (which wouldn't apply to the admin router) and apply some of the global link-building configuration such as URL romanization. This object is only created on demand, so you don't have the overhead of doing this until you need the object. To emphasize the point, even if we had no expectation that you wanted a particular type of object somewhere, you should be able to instantiate it quickly and correctly through the dependency injection container.
The specific app object that is instantiated varies based on the location being loaded; for example, public and admin pages have separate app objects which lead to different configurations and dependencies being setup. As an example, this changes the session configuration for each app so that admin sessions come from the DB while public sessions may come from a cache layer.
Generally speaking, one request will have one app object. While in an ideal world, you would be given the specific object/data that your code depends on. However, that's not always viable, particularly in add-ons. There are certain types of classes where we pass the app object in (such as services), but in a pinch, you can always access it via
\XF::app() and then pull the specific elements out that you need. This approach would fit more of a service locator pattern.
In most cases, container entries are represented as closures that are cached when initialized. This isn't always the case they; the container can store raw data or uncached closures. It also provides factory-style instantiation tools.
If this sounds complex, it's mostly because we're staying very abstract in the explanation. In most cases, your interaction with the app object may be something like
$app->db() to get the DB adapter out. You won't need to worry about what happens behind the scenes. However, if you are interested in manipulating the container, add-ons will be able to extend entries and change how they are instantiated. (This can even be done via config.php.)
Improved JS Framework
XF2 itself still uses jQuery as its primary JS framework but has removed its dependency on the aging and unsupported jQuery Tools. We have built our own framework around jQuery which makes instantiation and extension of JS easier.
We have two types of handler; a click handler and an element handler. Each handler type is an object which has similar behaviors and similar functions within that should be extended. Click handlers execute code once the element is clicked, and element handlers execute once the element is initialized. Let's take a look at a basic click handler:
JavaScript:
XF.LinkClick = XF.Click.newHandler({
eventNameSpace: 'XFMyLinkClick',
options: {
alertText: 'Link has been clicked!'
},
init: function()
{
alert('Initialization. This fires on the first click.');
},
click: function(e)
{
e.preventDefault();
alert(this.options.alertText); // Alerts: 'Link has been clicked!'
}
});
XF.Click.register('mylink', 'XF.LinkClick');
To set up an element to call the above code when clicked, you simply add a
data-xf-click attribute with a value of
mylink (the identifier, the first argument of the register line).
HTML:
<a href="javascript:" data-xf-click="mylink">Click here!</a>
Some interesting things to note:
- We specify a unique event namespace which means that a number of events will be fired when the link is clicked. Specifically, they are before the handler is initialized, after the handler is initialized, before the click code executes and after the click code executes.
- We have an options object. Options are passed to the handler automatically and allow an element to adjust its configuration based on
data-X attributes passed in. For example, when the above link is clicked, the alert "Link has been clicked!" will display. If we wanted this to alert something different, we can do that by adjusting the HTML as follows:
HTML:
<a href="javascript:" data-xf-click="mylink" data-alert-text="Something different!">Click here!</a>
Element handlers work in a nearly identical way, but the
init function for element handlers executes when the element is activated (usually on page load for the entire document, or on specific elements on demand).
It is also trivially possible to extend an existing handler. Let's take a look at a very basic element handler which when assigned to an element will change its text color to red, and underlined:
JavaScript:
XF.RedText = XF.Element.newHandler({
options: {
color: 'red',
decoration: 'underline'
},
init: function()
{
this.changeColor();
this.underline();
this.alert();
},
changeColor: function()
{
this.$target.css({
color: this.options.color
});
},
underline: function()
{
this.$target.css({
'text-decoration': this.options.decoration
});
},
alert: function()
{
alert('The text changed to red and underline...');
}
});
XF.Element.register('red-text', 'XF.RedText');
HTML:
<div data-xf-init="red-text">This is black text.</div>
When this element initializes, the "This is black text" text will turn red and become underlined and an alert 'The text changed to red and underline...' will be displayed on the screen.
We might want to re-use or extend this code for a different element handler, and this is possible using
XF.extend. This allows you to extend an existing handler with your own code. You can then change default options, or even override or extend entire functions with the added option of still calling the original function if desired. Let's see that in action:
JavaScript:
XF.BlueText = XF.extend(XF.RedText, {
__backup: {
'alert': '_alert'
},
options: $.extend({}, XF.RedText.prototype.options, {
color: 'blue',
decoration: 'line-through'
}),
alert: function()
{
this._alert();
alert('...actually, it changed to blue and strike through!');
}
});
XF.Element.register('blue-text', 'XF.BlueText');
HTML:
<div data-xf-init="blue-text">This is black text.</div>
What we have done here is we have registered a new element handler called
blue-text but the bulk of the behavior is coming from the
red-text handler.
Notice that the default options of the
red-text handler are overridden so that the text is going to be blue and line-through rather than red and underline.
We've also used a
__backup object to map the
alert function in the original handler to a new name of
_alert in the new handler. This is equivalent to extending a class and method in PHP and calling the
parent:: method.
The end result of this blue-text handler is that the color of the text is changed to blue and underline (based on our extended options), the alert text from the original handler is displayed ('The text changed to red and underline...'), followed by the new alert text we've specified in the new handler ('...actually, it changed to blue and strike through!').
While this demonstrates a clear hierarchy style, we have also exposed a more dynamic method of extension. This would likely fall into the realm of "monkey patching".
JavaScript:
XF.Element.extend('red-text', {
alert: function()
{
alert('Hey a different message');
}
});
This dynamically extends the
XF.RedText instance in place and overrides the functionality of the alert function. All code that is using the
red-text element will automatically use the extended version. Further, multiple extensions can happen to the same element handler without causing name conflicts; in this regard, it's similar to the class extension system for PHP.
Additionally, this
XF.Element.extend method can be called before or after the base element is registered. This is relevant when JS is loaded asynchronously. (Clearly, it needs to be called before the element is instantiated, but the JS framework won't do that until all pending asynchronous JS has loaded.)
Abstracted file system
Instead of writing to semi-hard coded paths, XF2's main file writes now write through an abstracted system. This makes it much simpler to write files out to locations other than a local disk. Commonly, this would involve writing the files to a cloud storage system such as AWS S3, but there are various other potential approaches.
Reading and writing is done through the file system object grabbed from the app (
$app->fs()). Here's what a path might look like:
Code:
internal-data://attachments/0/123-abcdef0123456789abcdef0123456789.data
This would identify the internal-data as the "mount" and the rest would represent any subsequent path and file name. The specific adapter would represent these as is appropriate for their usage.
The available mounts are:
- internal-data -- for data that is only accessible via PHP. By default, this is still the local internal_data directory.
- data -- data that is accessible via PHP and directly via a browser.
- code-cache -- this is specifically for generated PHP code that we then write out to files. By default this is within the internal_data directory. This is distinguished from regular internal data files because we will be including/requiring files via the specified path so that we can take advantage of opcode caching. You want accessing these to be fast.
- temp -- just for temporary files. This always resolves to a local temporary directory. This isn't used commonly but it can help transition from a potentially remote file to a file that is known to be locally accessible (as some processes require local files).
Phrase grouping
When we dynamically load phrases in XF1, if the phrase is not globally cached there can be a query overhead in terms of retrieving that phrase. Although globally caching is an option, it's often unnecessary to have something cached globally as it may only be used on certain pages.
Our solution for this in XF2 is phrase groups.
A phrase group is indicated by a special prefix applied to a phrase name in the format of
<group>.<phrase>. All phrases that belong to the same group are compiled and written out to a file on the file system for each language and loaded from there instead when needed.
Generic change logging system
In XF 1.3 we introduced a
user change logging system which causes all changes made by any user to any other user (or themselves) to be logged. The system itself is still present in XF2 and hasn't changed much on the surface, but behind the scenes, it is now handled by an entirely generic system which developers can extend for logging other changes to other content types besides users.
The change logging system is very simple to implement for your content types, too. You just extend our
AbstractHandler and configure the Entity/Entities to use the
ChangeLoggable behavior and the bulk of the logging and formatting of the logs happen automatically.
You can either configure an entire Entity to be "change-loggable", which means all fields by default will have changes logged, or you can opt into the logging on specific fields you explicitly want to log.
CLI Framework
This feature doesn't actually exclusively benefit developers, it also benefits site admins too, especially those with larger sites.
XF2 includes a framework which allows CLI commands to be defined and allow you to run those commands from a shell/command line. From a non-development point of view, it means admins of larger sites can perform intensive tasks such as site upgrades without having to worry about various timeouts that can be problematic when running such things through the web interface.
Upgrading the site using the CLI is not actually something new and it is something we actively recommend when we detect an upgrade of a larger site, but the framework behind it is generic, so it opens the door eventually for more tasks to be run in that way. As an example, however, here's what you would type from a shell on your server to upgrade XF to a new version:
Code:
C:\xf> php cmd.php xf:upgrade
Current version: 1051170
Upgrade target: 2000010 (2.0.0 Alpha)
Are you sure you want to continue with the upgrade? [y/n]
In addition to upgrading, you can also install XF2 from scratch, and upgrade your tables to support utf8mb4.
In terms of developers, there are some development commands which you may find useful. Here's a few of them:
xf-dev:addon-create This will take the initial steps necessary to create an add-on, and write out the initial addon.json file. The command asks various questions such as add-on ID and title and uses the responses to build and write out the file.xf-dev:addon-export <addon_id> Exports all of the add-on related data (phrases, templates, navigation etc.) to the [/ICODE]data.xml[/ICODE] file for the specified addon ID file.xf-dev:import --addon <addon_id> Imports all of the add-on related data (phrases, templates, navigation etc.) from the development output files on the file system to the database for the specified add-on.xf-dev:export --addon <addon_id> Exports all of the add-on related data (phrases, templates, navigation etc.) from the database out to files on the file system for the specified add-on.
In addition, it will be possible for developers to create their own commands for their own personal use or for distribution with add-ons.
Master/slave support in DB adapter
XF2 introduces master/slave replication via a new DB adapter which allows separate read and write connections to be made. By default, select statements will be sent to the read server and all other statements will be sent to the write server.
You can see a typical example of configuration below:
PHP:
$config['db']['adapterClass'] = 'XF\Db\Mysqli\ReplicationAdapter';
$config['db']['write'] = [
'host' => '192.168.10.1',
// ... username, password, dbname etc.
];
$config['db']['read'] = [
'host' => '192.168.10.2',
// ... username, password, dbname etc.
];
As well as inferring the correct connection to make from the type of query, you can also control the behavior by prefixing a query with a comment in the form of
-- XFDB=modifier.
Where the
modifier is one of:
fromWrite - forces a specific read query to come from the write server even if it normally wouldn'tforceAllWrite - forces this query and all subsequent queries to the write servernoForceAllWrite - if this query would normally force all subsequent queries to the write server, this option disables that (useful for a write you know won't be read back immediately or if it is, can tolerate lag)
General improvements
While we have called out a lot of new things in XF2, there are plenty of systems that have also changed under the hood but generally just represent improvements. These include:
- General reduction of static method usage. We now operate on object instances (where possible) except for some basic utility functions. This allows easier customization.
- Rebuilt permission compilation code which allows significantly more reuse for new content types that require permissions (such as resource or media gallery categories).
- Rebuilt BB code parsing system. This now separates rules defining the available BB codes, the parser, and the rendering mechanisms more cleanly. Significantly, BB code "processing" systems -- such as auto-linking, adding mentions, or limiting the allowed tags -- have now been split out separate analyzer and filterer systems, independent from the core rendering framework.
- Improved generic systems for prefixes and custom fields so that add-ons do not need to repeat as much boilerplate code to implement the basic functionality.
- A generalized connected accounts system to make it easier to support logging in/account creation through external services. We support Facebook, Google, Twitter, Microsoft, GitHub and LinkedIn out of the box, but we now include a powerful OAuth client library that includes Amazon, Dropbox, Instagram, Reddit, Spotify and others so add-ons should be able to add more connected account providers easily.
I think that sums of the most significant changes to the core of the XF2 code. Inevitably, there will be thousands of other changes that we haven't covered here and those will become more apparent once you start working with the code. We're still working very hard and marching towards the developer preview release.
And with that, we wish everyone a very happy new year!
