XF 2.0 Changing codeCachePath produces blank page

[jkilbride]

I'm running Xenforo in an AWS autoscaling group with the data and internal_data directories located on an EFS (NFS) mount. I'm working on upgrading our system from 1.5.21 to 2.0.7 and I noticed new code_cache and temp directories inside internal_data. I'm assuming that it would be better to put these on the local filesystem -- please correct me if I'm wrong. I have added the following to my config.php file:

// Put code_cache and temp directories in local /tmp
$config['codeCachePath'] = '/tmp/xenforo/code_cache';
$config['tempDataPath'] = '/tmp/xenforo/temp';

However, when I do this the forum and admin pages pull up blank with no errors anywhere. The /tmp/xenforo/code_cache directory is created along with an index.html file, so file permissions aren't a problem. I even manually changed the permissions to 777 (world read/write) and it didn't make a difference. The server is returning a 200 http code, even though the pages are completely blank:

172.19.0.1 - - [05/Jul/2018:18:48:15 +0000] "GET /index.php HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:61.0) Gecko/20100101 Firefox/61.0" "-"
172.19.0.1 - - [05/Jul/2018:18:48:45 +0000] "GET /admin.php HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:61.0) Gecko/20100101 Firefox/61.0" "-"

Notice the zero response length after the 200 response code. Am I missing some other configuration? Has anyone successfully changed the code_cache and temp file paths? I'm working with a freshly installed v2.0.7 inside a local Docker development environment using a Redis cache. Literally, no other changes - no posts, only the admin user, etc... Here's my full config file:

<?php

$config['db']['host'] = $_SERVER['DB_ENDPOINT'];
$config['db']['port'] = '3306';
$config['db']['username'] = $_SERVER['DB_USER'];
$config['db']['password'] = $_SERVER['DB_PASS'];
$config['db']['dbname'] = $_SERVER['DB_NAME'];

// Caching with Redis
$config['cache']['enabled'] = true;
$config['cache']['provider'] = 'Redis';
$config['cache']['config'] = [
    'host' => $_SERVER['REDIS_HOST'],
    'port' => $_SERVER['REDIS_PORT'],
    'database' => $_SERVER['REDIS_DB']
];
$config['cache']['sessions'] = true;

$config['fullUnicode'] = true;

// Put code_cache and temp directories in local /tmp
$config['codeCachePath'] = '/tmp/xenforo/code_cache';
$config['tempDataPath'] = '/tmp/xenforo/temp';

Everything works correctly when I remove the codeCachePath and tempDataPath settings. Any help / insight would be appreciated. Thanks!

 

[Chris D]

Changing the temp location won’t affect anything but code_cache is where compiled templates and other things live. So unless you copied the original contents of code_cache over to the new location, this is expected.

The quickest solution is to rebuild master data by going to /install which will recompile everything to the new location.

 

[jkilbride]

@Chris D Thanks for the quick response!

I went to /install and used the Rebuild Master Data option. It's working now. As it turns out, it looks like there were a couple of errors being reported in the admin, but not in the server / PHP logs. I couldn't see them, because the admin was blank. Is there a way to see errors like this without having to login to the admin, possibly from the CLI?

Couple of questions:

- My new production system uses a 3rd party style and has lots of tweaks. Is it safe to run "rebuild master data" on a highly customized installation?

- I have multiple instances running off the same data and internal_data directories. Will it cause problems if the code_cache contains different info on the different instances?

- I haven't been able to find much information on how to setup Xenforo 2.x in an HA environment. Is there a guide somewhere for what files should be shared across servers? Right now, I am sharing the full data and internal_data directories. It works, but it kind of feels like overkill. Am I doing it right?

Also, I would recommend adding the rebuild master data step to the "Config.php Options" section of the Xenforo 2 manual. It's not obvious that this needs to be done after changing the codeCachePath setting.

 

[jkilbride]

Ok, I can see some problems with having code_cache on the local drive in an HA setup. Right now, I'm using Docker to run my dev environment. Each time I stop / remove my containers, the /tmp directory inside the php-fpm container disappears and I have the same problem of a blank page when I bring the containers back up again. I can solve this in Docker by mapping the /tmp directory in the container to the host drive, but I have a feeling this is going to be a problem in an HA environment when a new instance is deployed. The local code_cache directory won't exist and, apparently, won't be re-created without running Rebuild Master Data. So, the answer to question two above is "yes" - there will be problems.

Do you guys have any suggestions for how to handle this? I don't like the idea of running the code_cache on an NFS mount. Is there a baseline set of files that need to exist in code_cache for it to work, that I could copy over to a new deployment? I'm tempted to do a fresh install and look at it before loading any pages in my web browser, but would prefer a definitive answer. Otherwise, can this cache be moved to Redis? How do other people run XF 2 in a multi-server environment?

 

[jkilbride]

So.... no pointers, guides, or other information for running this thing in a multi-server environment? Do I have to file a ticket to get an answer on this?

 

[Chris D]

Do you guys have any suggestions for how to handle this? I don't like the idea of running the code_cache on an NFS mount. Is there a baseline set of files that need to exist in code_cache for it to work, that I could copy over to a new deployment? I'm tempted to do a fresh install and look at it before loading any pages in my web browser, but would prefer a definitive answer. Otherwise, can this cache be moved to Redis? How do other people run XF 2 in a multi-server environment?

There’s no base line set of files, the files consist of compiled templates and phrases and so the contents can and will change depending on updates to your style and phrases.

I don’t have any specific advice on setting up a multi-server environment other than in that environment you need to ensure that the files are available to each instance and be 100% consistent across each instance.

Whether that means you have a single mount which all instances read from / write to (but would represent a single point of failure) or you maintain multiple instances of the data replicated across each server is entirely up to you, but consistency of the data is the most important thing. The data shouldn’t change frequently though, it should only change when templates and phrases are changed.

 

[jkilbride]

@Chris D Ok, thanks. I'm not worried about having a single point of failure, because Amazon EFS is fault tolerant. I'm more worried about the speed of access to the files, especially in code_cache. NFS is orders of magnitude slower than a local drive. However, I just ran across cachefilesd, which is an NFS caching daemon that uses the local disk to cache frequently accessed files. So, I'm going to give that a try and see what happens.

I'm sorry to go further off-topic from my original question, but could you point me to an example of fetching data from an HTTP api endpoint and storing / retrieving that data with Redis? I've been through the dev docs and didn't see anything. I've searched a bit on the forums, but it's taking quite a while to wade through all the messages. A quick example is all I need. Any pointers would be appreciated.

 

[Chris D]

I'm sorry to go further off-topic from my original question, but could you point me to an example of fetching data from an HTTP api endpoint and storing / retrieving that data with Redis? I've been through the dev docs and didn't see anything. I've searched a bit on the forums, but it's taking quite a while to wade through all the messages. A quick example is all I need. Any pointers would be appreciated.

There is no existing endpoint for such things as we don’t have any sort of REST API in the software.

If this is related to the same thing there is actually a third party adapter you might be able to use for storing stuff like the code cache in Redis.

Our internal file system is abstracted using a third party library called Flysystem and although we only include the local storage adapter by default, other adapters are available. I’ve just spotted a few Redis based Flysystem adapters on GutHub so you may actually be able to come up with something to offload the code cache to that.

 

[jkilbride]

That's an interesting idea, but not what I was talking about. Sorry for not being more clear.

I have my own backend API that I need to fetch some data from and then I want to store it in the local Redis cache. I already have Redis working, I just want to access it from my own code. Actually, what I'm looking for is something like this for the http stuff:

$client = \XF::app()->http()->client();

I just came across this in another thread. Is there any documentation available on the \XF::app() object? I thought I saw an example of accessing the cache from the app() object, too, but I can't find that thread.

As for the Flysystem stuff, I've already looked at your examples for moving the "data" directory to an AWS S3 bucket. Moving code_cache, or even the entire internal_data directory, to Redis would solve a lot of the multi-server issues I'm running into, right now. Can you point me to more info on the $config['fsAdapters'] structure? Is there a matching "internal_data" key for that array, like there is for "data" in your S3 example?

 

[Chris D]

As for the Flysystem stuff, I've already looked at your examples for moving the "data" directory to an AWS S3 bucket. Moving code_cache, or even the entire internal_data directory, to Redis would solve a lot of the multi-server issues I'm running into, right now. Can you point me to more info on the $config['fsAdapters'] structure? Is there a matching "internal_data" key for that array, like there is for "data" in your S3 example?

Last thing first, yes, there's a matching key but it's named internal-data and then another for code-cache. Worth noting, actually, before I forget to mention it, there's this stipulation with the code-cache override in our code:

// if you override this, codeCachePath must still contain the code as we will use it to include files

I didn't actually work on the file system stuff, but this seems to suggest that the files will need to be stored in that path too, so I'm not entirely confident offloading it somewhere else is going to solve the problem. But it makes sense, from the context of how PHP works; the contents of the code cache directory consist primarily of PHP scripts so when we use include or require to call those it is not going to work unless they're accessible locally. That might put a dampener on the idea of offloading that, but of course similar issues won't exist for data/internal_data.

On the subject of the code example for the AWS S3 bucket and the other file system mounts we have, and how to override them, take a look at src/XF/FsMounts.php which is where we apply those config.php fsAdapter changes.

That's an interesting idea, but not what I was talking about. Sorry for not being more clear.

I have my own backend API that I need to fetch some data from and then I want to store it in the local Redis cache. I already have Redis working, I just want to access it from my own code. Actually, what I'm looking for is something like this for the http stuff:

$client = \XF::app()->http()->client();

I just came across this in another thread. Is there any documentation available on the \XF::app() object? I thought I saw an example of accessing the cache from the app() object, too, but I can't find it that thread.

The App object is our dependency injection container. It's a fairly monolithic object that can be both global and app type specific (app types being public, admin, install etc. depending on which part of the system you're using).

The cache provider is accessible using:

$cache = \XF::app()->cache();

This will return a Doctrine CacheProvider object based on which cache provider you have configured in config.php and the manual describes how to configure Redis here: https://xenforo.com/xf2-docs/manual/cache/#redis

We use the cache for things like session storage, CSS cache etc. That actually provides a good example of how you might be able to use the cache provider to store whatever data you need to store. We pass in the aforementioned Doctrine CacheProvider object and you can find examples of how we read from the cache XF\CssRenderer::getFinalCachedOutput() and write to the cache XF\CssRenderer::cacheFinalOutput() though, really, an even simpler example is:

$cache = \XF::app()->cache();

$cache->save('some_id', 'some_data');

$data = $cache->fetch('some_id'); // contains 'some_data'

If you want to poke around in the HTTP client stuff, the code for that is actually stored in what we call a sub-container which is an App object which we use to logically group similar containers. You can see all the code for that in src/XF/SubContainer/Http.php. Though the API for the HTTP client will be familiar if you've ever used Guzzle as that's the underlying library we use.

 

[jkilbride]

@Chris D Thanks! Plenty to look at here. These examples should be enough to get me going.

 

[PaulB]

Whether that means you have a single mount which all instances read from / write to (but would represent a single point of failure) or you maintain multiple instances of the data replicated across each server is entirely up to you, but consistency of the data is the most important thing.

This means opcache needs to be disabled in any multi-server scenario. Ouch. It also rules out the possibility of using NFS even with noac and sync.

In practice, as long as only a single server is performing writes to the code cache (read: as long as you only use the CLI for significant changes), it's unlikely that any serious issues will arise. With any sort of caching--including opcache--reading servers will likely return errors until their caches expire as they'll be seeing a combination of stale and up-to-date data, but that won't usually result in long-term issues. That being said, it's possible for something awful to happen; the exact behavior is going to be different every time. It's a game of chance.

Consistent, highly-available, and partition-tolerant: pick two.