Granular filesystem adaptor

digitalpoint

Well-known member
I found a little time to start building an add-on today that seems like it will do what I'm talking about. It maintains backward compatibility like so...

XenForo's default 3 adapter overrides can still be used like normal:

PHP:
$config['fsAdapters']['data'] = 'Your\Adapter::getAdapter';
$config['fsAdapters']['internal-data'] = 'Your\Adapter::getAdapter';
$config['fsAdapters']['code-cache'] = 'Your\Adapter::getAdapter';

Except now instead of allowing exactly 3 adapters in the mount manager (which is \XF::fs()), you can have optional "sub-adapters". For example, if your config was like so:

PHP:
$config['fsAdapters']['internal-data'] = 'Your\Adapter::AwsS3';
$config['fsAdapters']['internal-data/image_cache'] = 'XF\FsMounts::getLocalAdapter';
$config['fsAdapters']['internal-data/keys'] = 'DigitalPoint\FileSystem\Adapter\DataRegistry::getAdapter';

So in that example, the 3 existing allowed adapters become "defaults" for their respective locations and appending a directory to the default name allows you to override whatever the default is for just that directory. So right there, you could have internal-data stored in AWS S3, the image_cache within there is kept in the local file system and your DKIM key stored in the data registry (that's already an Adapter I created, although I haven't tested it yet).

It works without having a defined default as well... so if someone wanted to JUST move their DKIM key to the registry, you can just add that one line and everything else stays in the default local file system.

So far it's only taken a single XenForo class extension (and only altering a single method in that class).

There's more to do, but that's how I have it working so far. If anyone has any ideas, feel free to toss them in.
 
So far it's only taken a single XenForo class extension (and only altering a single method in that class).

There's more to do, but that's how I have it working so far. If anyone has any ideas, feel free to toss them in.
That's cool - looking forward to see full code as I still don''t get how this would work, lol.

You can of course mount another adapter as internal-data/keys by extending \XF\FsMounts::loadDefaultMounts, but files accessed as internal-data://keys/emailDkim*.key would not use this adapter - they would still use the adapter at internal-data (at least in my tests).
 
That's cool - looking forward to see full code as I still don''t get how this would work, lol.
Magic. 😉

You can of course mount another adapter as internal-data/keys by extending \XF\FsMounts::loadDefaultMounts, but files accessed as internal-data://keys/emailDkim*.key would not use this adapter - they would still use the adapter at internal-data (at least in my tests).
Ya. Doing it that way is where you would lose backward compatibility and start down the road of changing everywhere that internal-data:// is used. That’s too much work imo (you would also need to keep up with new locations down the road).
 
For the most part, ya... it would work as it works now, but it wouldn't show things in sub-directories you put in a different adapter. So in my above example if internal-data was on S3, image_cache was local and keys were in the data registry, you would get a list of the stuff specifically in the S3 adapter (everything except image_cache and keys).

If someone is using the abstracted file system to manage files/folders in multiple directories in one go (as in not using a path, only a prefix), it wouldn't work great for that.

Now if you actually had a path on that command, yes it would work just fine. For example:
PHP:
$files = \XF::fs()->listContents('internal-data://image_cache/', true);
 
Last edited:
If someone is using the abstracted file system to manage files/folders in multiple directories in one go (as in not using a path, only a prefix), it wouldn't work great for that.
That's what I epxected, so this would break existing functionaly (I do use such calls in some Add-ons) to some extend.

So now were back at:
Unless I am missing smth. or you suggest to implement an OverlayFS-type Flysystem Adapter I don't think it is possible to implement this suggestion without breaking backwards compatibility
?
 
It wouldn't be terribly difficult to implement with the way I set it up (in fact the mount manager would already have all the info it needs for use in the listContents() call already). Probably not something I'm going to implement though as there's nowhere in XenForo that uses listContents() without a path that I'm aware of.

The good news is I've already set it up to use XenForo's normal classExtend method, so someone could simply extend the listContents() if they wanted such functionality (again, the data it would need to do it is already available to that method, so you wouldn't need to do any trickery with passing extra data into it or anything).

listContents() is never going to be a 100% reliable way to get contents of directories... Some Flysystem adapters don't support listContents() at all (say you were storing data in a write-only FTP server, or as an in-memory adapter as an example). So really you can't even be absolutely sure a user is using an adapter that even supports listContents() at all.

As an example, the DataRegistry adapter I made doesn't support listContents(), because there are no "directories" to list. It simply returns an empty array if someone were to try and get a directory on it.

PHP:
public function listContents($directory = '', $recursive = false)
{
return [];
}
 
listContents was just an example.

Would
PHP:
$fs = \XF::fs()->getFilesystem('internal-data');
$fileContents = $fs->read('/keys/emailDkim-L7ekmvWZ7H.key');
work or any other call on the filesystem with path starting with /keys/
?
 
listContents was just an example.

Would
PHP:
$fs = \XF::fs()->getFilesystem('internal-data');
$fileContents = $fs->read('/keys/emailDkim-L7ekmvWZ7H.key');
work or any other call on the filesystem with path starting with /keys/
?
It would, yes. Whole thing would be kind of pointless if you couldn't do the most basic read/writing with it, no? :)

The logic for that would be as follows...

We have a path with a directory, let's see if $path . '/' . $firstDirectory exists as a different prefix (internal-data is a file system "prefix" as an example). If that is a defined prefix (because someone added it to the config.php file)...

Remember this?
PHP:
$config['fsAdapters']['internal-data/keys'] = 'DigitalPoint\FileSystem\Adapter\DataRegistry::getAdapter';

...if it exists, use that prefix/adapter. If it doesn't exist, then just use internal-data like normal.

The important thing here is it's looking at a combo of prefix (internal-data PLUS the first directory in the path to find an "alternate" adapter transparently).

And that's how we are maintaining backward compatibility.

Also, your example can be shortened to this and it works just as well...

PHP:
$fileContents = \XF::fs()->read('internal-data://keys/emailDkim-L7ekmvWZ7H.key');
 
Last edited:
So I actually got around to testing it... left everything as default (local file system in this case), but overwrote just the keys folder to use the DataRegistry adapter I made and things seem to be working as expected. internal-data://file_check/file-check-xxx.json is still accessible via local file system (didn't break what was already there). Writing the file internal-data://keys/a-test-key.key, and it doesn't exist in the local file system, but does exist in the xf_data_registry table. Reading the same file back works, as does deleting it.

Making sure it doesn't add database queries when reading the key (I have registry entries stored in Memcache), and it does not add any queries to xf_data_registry.

...so I'd say it's working more or less. Probably should do more testing, but it's a good sign. 🤷🏻‍♂️

So that solves the issue of how DKIM key is stored. It also allows you to use any Flysystem adapter (for example AWS S3) for specific folders within data or internal-data without fully changing the adapter in an all-or-nothing fashion. It also maintains backward compatibility (don't need to recode everything that uses \XF::fs(). Seems like a win to me.
 
So that solves the issue of how DKIM key is stored.
I am already excited to see this in action, as the real probelm with DKIM keys (apart from being stored in internal-data) seems to be taht they are being read for every single email (instead of being read just once per script call).

Besides that I still don't understand how you were able to achive that (without building an Overlay-FS type adapter - either directly or by using *.before Plugins to redirect calls).
 
I agree that it's still not ideal if it's reading the DKIM key multiple times if a single process is sending multiple emails. But at least using the data registry multiple times isn't as bad.

Making 10 requests to Memcached to send 10 emails is much better than making 10 requests for the same object in an S3 storage bucket.
 
Making 10 requests to Memcached to send 10 emails is much better than making 10 requests for the same object in an S3 storage bucket.
Yes, absolutely - but as said before, a good approach woud be to just use the existing DI container to load the key only once.
Heck, that's what dependency injection does exist for - doesn't it? :)
 
Want to know something I realized today that makes reading keys for every email even more gross?

The way XenForo uses it's abstracted file system (Flysystem), it read two different times for each underlying \XF::fs()->read() call. By default Flysystem asserts that a file exists before it does whatever action you are actually wanting to do. This means every time you read the DKIM key (or anything with the filesystem adapters), it first stats the file (if you are using the default local filesystem) with a file_exists() call the file to see if it's there. If it is there, then it goes back and actually reads the file.

This means if you were sending 20 emails, you would not only read the DKIM key 20 times, but you would also stat the file 20 times as well.

For other filesystems, it's a bigger issue... for example if you were using S3, you need 2 backend API calls each time you read a file (one to check if the file exists, and then another to read it).

I've ended up making some additional changes so any adapter can disable the assert check. So for my R2 adapter and also the data registry adapter, you've just cut the reads/API calls in half regardless of what you are doing.
 
I made a simple add-on that allows you to store just the internal_data/keys folder in the XenForo data registry.

 
Top Bottom