XF 2.2 Unit Testing with XenForo (and other end-to-end testing)

[robdog]

I am sure this has been discussed before and I have seen an addon around this topic, but I wanted to get some other ideas/thoughts on how other devs are unit testing their code or if folks are performing end-to-end testing within their XenForo app. I am really curious on the set up and tear down. Do you use mock data? How do you testing entities and repositories?

Anyways, just curious what people do, thanks!

 

[Sim]

I'm sure you've already seen it - but for reference in case other people have not yet seen it, here is my tutorial on unit testing XenForo addons: https://xenforo.com/community/resources/unit-testing-xenforo-addons-tutorial.7508/

I explicitly mention that my tutorial only covers the "unit testing" side of things - not functional tests or feature tests or end-to-end tests or whatever else you like to call it when you test a full HTTP request/response cycle. I'm going to refer to these as "feature tests" - because that's what Laravel calls them.

Feature tests are where you simulate a browser HTTP request to a certain route using a HTTP GET or POST commands which exercise entire paths through your application and examine the response, setting expectations on things that should have occurred during execution.

After thinking about it for a while while writing that original tutorial, I concluded that attempting to do feature testing in XenForo - as it is currently architected - was too difficult, mainly due to the bootstrapping process.

When feature testing, you need to inistantiate an almost fully-hydrated XenForo install so that you have data in the system to test against.

When you think about how long it takes to install a new instance of XenForo and how much data is pre-populated into the database during install (tempates, phrases, permissions, default nodes, admin user, etc) - it is a fairly significant process. Take a look at the database after performing a fresh install - it's already got a lot of data in it.

More importantly, to properly feature test, you likely need to do this process for every single test - which is going to take a long time, especially if using an on-disk database like MySQL.

When testing Laravel applications, I will use SQLite database operating in-memory - so all database operations are ephemeral and fast. I also only bootstrap the parts of the application that are required for testing.

I did investigate whether it would be possible to write an SQLite adapter for XenForo to use for testing purposes, but I believe that there are some MySQL-only functions used by XenForo which would make this quite difficult.

So between the lack of SQLite support and the challenges of bootstrapping a working XenForo install with sufficient data in it to be able to perform feature tests - I put it in the too hard basket.

I guess you could perform feature tests against an existing XenForo install - but the issue there is repeatability. If you aren't discarding your application after every test and re-instantiating it for the next test, you're not going to be testing the exact same application every time. Which means that multiple executions of your tests could lead to different outcomes making reliable testing difficult and automated testing pretty much impossible.

Laravel also provides useful tools for mocking data by way of factories - classes that extend their models and provide a set of attribute values using the Faker PHP library. This makes it really easy to quickly populate your database with meaningful fake data for testing purposes.

If attempting to do feature tests on XenForo - this Faker library is probably the first thing I would investigate for providing mock data.

I'd be very interesting to hear if any other devs have gone down the path of creating a feature test system and how they approached it.

 

[Xon]

For most application code;
A depressing amount of common unit tests are redundant once you have correctly type-hinted code and use static analysis, this includes phpstorm's inspector tooling.

IMO, Most unit tests (ie testing individual functions) are almost entirely worthless, and integration testing is a lot of work. That said, there is no silver bullet but using some automated testing is often more useful than none or too much.

For libraries, yes please unit test. Ensure those API boundaries are stable and keep doing what they where doing before.

Why different ideas testing for application vs library code? Scope.

A library will have an API boundary of some sort and a fairly limited scope, and the scope is generally under control of the developer.

Application scope is nearly entirely "business logic" driven, and nailing down a business process is extremely time consuming and complex task. In comparison, application logic is often at the whim of regulatory bodies and other entities vastly larger than you (tax, google SEO, etc)

I guess you could perform feature tests against an existing XenForo install - but the issue there is repeatability. If you aren't discarding your application after every test and re-instantiating it for the next test, you're not going to be testing the exact same application every time. Which means that multiple executions of your tests could lead to different outcomes making reliable testing difficult and automated testing pretty much impossible.

If you need to test the entire stack, using something like ZFS snapshots and then starting a mysql instance against a clone/snapshot is probably the fastest way for a feature set testing against a full mysql instance.

If you write you tests to operate on threads/forums at a time, you can largely avoid mutating parts you don't expect. And if they do mutate that would be a bug to catch.

But one thing you need to be 100% aware of when building a test suit is this;

• Flaky test make the entire thing actively useless.

Test reliability is extremely important.

 

[Sim]

IMO, Most unit tests (ie testing individual functions) are almost entirely worthless, and integration testing is a lot of work. That said, there is no silver bullet but using some automated testing is often more useful than none or too much.

For libraries, yes please unit test. Ensure those API boundaries are stable and keep doing what they where doing before.

Yes, I've found my primary goal for unit testing is regression testing on version upgrades.

I want to be confident that after upgrading the version of XF, or moving to a new version of PHP, that my code runs exactly the same way it did before.

Occasionally if I have some particularly complex logic, I find it useful to write unit tests to validate that my function does what I expect it to.

Frequently I find when writing unit tests that it exposes weaknesses in my code - well written code should generally be quite testable, and if I'm having trouble writing tests for it, then I usually find that re-architecting things to make it testable make it fit together more reasonably.

But testing that a function which does something obvious, always returns that obvious thing - is a bit pointless.