XF 2.1 XenForo, Laravel, and PHPUnit: XenForo Dev Environment, Factories, and Cleanup?

[improbable_airtime]

I've adopted a TDD workflow the past few months and obviously since Laravel comes with PHPUnit and a whole testing suite working out the box, this is pretty neat.

But I need to work on an app that interacts with XenForo in a variety of different ways, most of which will be accomplished by including the XF.php file and using classes and functions directly.

However, one thing I've been struggling to figure out in my head is how exactly I can approach this TDD workflow when I rely on structures/data from XenForo?

For example, a bit of an explanation/pseudo.

// My Laravel app has it's own "User" model that has a column for a users forum_id.
// I don't particularly want to mock it, and I'd like to test with a variety of different users.
// My solution to this would be to make my own custom "Factory" that creates new XF users via `XF.php`
// (Even the above I'm unsure if it's possible, but I'm sure I could get it to work somehow.
// But then comes "cleanup". After a test is run, that user that was created should be deleted and should no longer exist.
// Same concept with various other things like; Threads, Comments, etc... Not really sure how to accomplish this "cleanup".

I'm not sure if this is really within the scope of XF specifically, but I'd like to hear if anyone else has been in a similar situation to this and has got a decent setup. I'm obviously running a local version of XF on my machine for dev purposes so I have no concerns for "possible data corruption" forum wise or anything like that. This is purely for me to generate new forum users, posts, etc... And assert against them, then they are no longer needed and can be deleted.

Hopefully someone can provide a solution!

 

[Sim]

I'm currently developing a unit testing framework for XenForo based loosely on Laravel's testing model.

Essentially, I'm building a base PHPUnit TestCase class which does most of the heavy lifting for you, allowing you to selectively swap out components of XenForo with mocks that you can use for testing.

So far I can mock the following:

• arbitrary items in the container and in subcontainers, as well as factories
• repositories, finders and read-only entities (see notes below) via the entity manager
• errors and exception logging
• phrases
• options (not actually mocked - just some utilities to directly over-ride options for testing)

Note that my focus so far has been unit testing, not functional testing - so I've been only focusing on isolated bits of logic contained primarily in custom containers and subcontainers, repositories and service classes that I use within my own addons.

To be clear - if all of your logic is in controllers, this framework won't really help you much.

Just a note on entities - mocking read-only entities is pretty trivial (although cumbersome, given they typically have a lot of function calls for getting/setting values).

However, given that the save() method on entities is marked final, we cannot effectively mock that - so a different approach is required for entities which need to be persisted back to the database.

Laravel's approach is to not mock Eloquent models (which are the rough equivalent of XenForo's entities), but to instead interact with the actual database itself and it gives you some testing utilities to test for data that exists (or does not exist) in the database.

However, there are two limitations here:

First, XenForo currently only has a MySQL driver, which introduces two other issues for us - it's relatively slow to write to an actual database (because it generally wants to persist things to disk) - and then you need to explicitly clean up after yourself, which adds even more time and complexity. I get around this with Laravel apps by using an in-memory SQLite database which gets rebuilt for every test. It's quick because it's not writing anything to disk. I've yet to start looking into the feasibility of writing an SQLite adapter for XenForo.

The second issue with database testing is that a fully functional XenForo application relies on data that has been installed in the database before you even create any content or users. It also takes quite a while to "build" that data during install - although I would hope that an in-memory database would be somewhat quicker.

Given that we ideally want to effectively build and then destroy the database for every single test (one of my addons has 54 separate tests so far, and I'm probably only half done), we need this to happen quickly and reliably - and more importantly, it needs to be automated.

I haven't started down the path of looking at what is involved with automating the installation of XenForo data for every test - I'd need to create the SQLite database adapter first before I look at that.

We would also need to automate the installation of your addon so it can actually be tested.

Once that is all in place though, I would think it would be fairly straight forward to build some factories to insert users / threads / posts etc into the database - we can utilise the XenForo services for most of those things.

Of course, the alternative approach is to just use the actual MySQL database used by your development XenForo server for our testing - but you'd need to be very careful about how you insert and remove data for testing purposes and you couldn't necessarily rely on assumptions about what is already there (because the act of testing will necessarily change it!).

For now, I might just look at mocking the database adapter itself and just check that queries were run - will get cumbersome to test the actual queries that were executed though, because they can get complicated.

Other things I still need to do:

• implement a version of Laravel's HTTP testing framework which allows you to make HTTP requests to your application and then examine the output. This is much more about "functional" or "feature" testing than unit testing though, and so isn't high on my priority list right now. It will be useful for testing APIs though.
• look into testing console command input / output - again, not really a high priority at this point
• some helpers to mock the cache
• helpers to test the registry
• mail testing
• filesystem testing
• job testing

Given the nature of my addons, mail testing and job testing are going to be fairly high up on my list of priorities.

I do plan on releasing an early version of this library soon - it is a Composer package you can install into your addon (require-dev so it's only there during development!!) and then you just use my base TestCase class for your PHPUnit tests and it does all the magic for you. It won't have any impact on your production code - I'll include a build.json file which removes the test code from your addon before it gets packaged up for deployment.

If anyone has written an SQLite database adapter for XenForo (or is interested in doing so!), I'd love to talk to you!

 

[Liam W]

If anyone has written an SQLite database adapter for XenForo (or is interested in doing so!), I'd love to talk to you!

It'd be pretty difficult (read: impossible) to write an adapter for a different DBMS, because despite appearing DBMS agnostic, there are quite a few direct queries that rely on the MySQL DSL.

It might be possible to use for limited testing, however.

 

[Sim]

It'd be pretty difficult (read: impossible) to write an adapter for a different DBMS, because despite appearing DBMS agnostic, there are quite a few direct queries that rely on the MySQL DSL.

It might be possible to use for limited testing, however.

Yeah, I found the same issue with Laravel - it's nice having a framework that allows you to swap out database adapters at the configuration level - but the reality is that there are always going to be some DBMS-specific niceties you'll likely want to use. If you changed your DBMS, you would likely rewrite parts of your code to optimise for that specific DBMS - so it's not transparently swappable.

I recently had that problem when writing unit tests for one of my Laravel apps - one of the queries I had written was optimised for MySQL and couldn't be tested on SQLite. To make it worse - it was one of the most critical parts of the application, so testing the logic that came before it was really important.

In this case, because performance was critical, I ended up completely re-writing the logic and used Redis to temporarily store the data before persisting it to the database in a separate task.

 

[improbable_airtime]

I'm currently developing a unit testing framework for XenForo based loosely on Laravel's testing model.

Essentially, I'm building a base PHPUnit TestCase class which does most of the heavy lifting for you, allowing you to selectively swap out components of XenForo with mocks that you can use for testing.

So far I can mock the following:

• arbitrary items in the container and in subcontainers, as well as factories
• repositories, finders and read-only entities (see notes below) via the entity manager
• errors and exception logging
• phrases
• options (not actually mocked - just some utilities to directly over-ride options for testing)

Note that my focus so far has been unit testing, not functional testing - so I've been only focusing on isolated bits of logic contained primarily in custom containers and subcontainers, repositories and service classes that I use within my own addons.

To be clear - if all of your logic is in controllers, this framework won't really help you much.

Just a note on entities - mocking read-only entities is pretty trivial (although cumbersome, given they typically have a lot of function calls for getting/setting values).

However, given that the save() method on entities is marked final, we cannot effectively mock that - so a different approach is required for entities which need to be persisted back to the database.

Laravel's approach is to not mock Eloquent models (which are the rough equivalent of XenForo's entities), but to instead interact with the actual database itself and it gives you some testing utilities to test for data that exists (or does not exist) in the database.

However, there are two limitations here:

First, XenForo currently only has a MySQL driver, which introduces two other issues for us - it's relatively slow to write to an actual database (because it generally wants to persist things to disk) - and then you need to explicitly clean up after yourself, which adds even more time and complexity. I get around this with Laravel apps by using an in-memory SQLite database which gets rebuilt for every test. It's quick because it's not writing anything to disk. I've yet to start looking into the feasibility of writing an SQLite adapter for XenForo.

The second issue with database testing is that a fully functional XenForo application relies on data that has been installed in the database before you even create any content or users. It also takes quite a while to "build" that data during install - although I would hope that an in-memory database would be somewhat quicker.

Given that we ideally want to effectively build and then destroy the database for every single test (one of my addons has 54 separate tests so far, and I'm probably only half done), we need this to happen quickly and reliably - and more importantly, it needs to be automated.

I haven't started down the path of looking at what is involved with automating the installation of XenForo data for every test - I'd need to create the SQLite database adapter first before I look at that.

We would also need to automate the installation of your addon so it can actually be tested.

Once that is all in place though, I would think it would be fairly straight forward to build some factories to insert users / threads / posts etc into the database - we can utilise the XenForo services for most of those things.

Of course, the alternative approach is to just use the actual MySQL database used by your development XenForo server for our testing - but you'd need to be very careful about how you insert and remove data for testing purposes and you couldn't necessarily rely on assumptions about what is already there (because the act of testing will necessarily change it!).

For now, I might just look at mocking the database adapter itself and just check that queries were run - will get cumbersome to test the actual queries that were executed though, because they can get complicated.

Other things I still need to do:

• implement a version of Laravel's HTTP testing framework which allows you to make HTTP requests to your application and then examine the output. This is much more about "functional" or "feature" testing than unit testing though, and so isn't high on my priority list right now. It will be useful for testing APIs though.
• look into testing console command input / output - again, not really a high priority at this point
• some helpers to mock the cache
• helpers to test the registry
• mail testing
• filesystem testing
• job testing

Given the nature of my addons, mail testing and job testing are going to be fairly high up on my list of priorities.

I do plan on releasing an early version of this library soon - it is a Composer package you can install into your addon (require-dev so it's only there during development!!) and then you just use my base TestCase class for your PHPUnit tests and it does all the magic for you. It won't have any impact on your production code - I'll include a build.json file which removes the test code from your addon before it gets packaged up for deployment.

If anyone has written an SQLite database adapter for XenForo (or is interested in doing so!), I'd love to talk to you!

Glad to hear you are working on this. I'm mostly in need of unit-testing and factories to ensure my functionality works. However, since you are still working on it would my best bet be to just mock the responses I need?

The worst part I think is the cleanup. Even if it's MySQL and not SQLite, it's still a pretty big task to keep track of all that data and then discard all of it. Not to mention it will require another connection to a separate database on-top of the Laravel one.

 

[Sim]

To be clear: my unit testing framework is for testing XenForo addons.

It won't help you with your Laravel app - that already has a unit testing framework.

I don't understand enough about what your Laravel app is doing to be able to make suggestions on how to proceed.

 

[Sim]

Note that I have completed the first phase of development for my XenForo Unit Test Framework and published a tutorial on how to unit test XenForo addons - Unit testing XenForo addons - tutorial

Composer package is here: hampel/xenforo-test-framework - BitBucket

Still lots of work to do - but it's a good start and I'm actively using it to unit test my own addons successfully.