Flysystem adapter implementations for cloud storage

[Sim]

It would be useful to provide a couple of flysystem adapter implmeentation for popular cloud storage providers (eg AWS S3 / Rackspace) so that people can confidently use them out of the box without relying on 3rd party addons (regardless of how simple those addons may be to implement).

It's quite important to have something we can use which is fully tested and supported - especially for such a fundamental feature which has the potential to break large parts of our site if it goes wrong.

 

[Kirby]

Hmm, maybe I don't undertand your suggestion but those adapters are available from the vendor?
https://github.com/thephpleague/flysystem-aws-s3-v3
https://github.com/thephpleague/flysystem-rackspace

 

[Sim]

Hmm, maybe I don't undertand your suggestion but those adapters are available from the vendor?
https://github.com/thephpleague/flysystem-aws-s3-v3
https://github.com/thephpleague/flysystem-rackspace

Yes, but you still need to implement an XF2 addon to be able to use those adapters.

Not everyone as the skills to be able to do that, and not everyone will be happy to use a 3rd party addon which implements those adapters because of support issues.

 

[Kirby]

I haven't actually tested it, but I don't think you'll need an Add-on, just a config:

$config['fsAdapters'] = [
    'data' => function () {
        return new \League\Flysystem\Filesystem(
            new \League\Flysystem\AwsS3v3\AwsS3Adapter(
                new \Aws\S3\S3Client([
                    'credentials' => [
                        'key'    => 'your-key',
                        'secret' => 'your-secret'
                    ],
                    'region' => 'your-region',
                    'version' => 'latest|version',
                ]),
                'your-bucket-name'
        ));
    },
];

and the adapter itself including it's dependencies.

 

[Sim]

I don't think you'll need an Add-on ... [just] the adapter itself including it's dependencies.

How do you propose to get the adapter itself into a location where the XF core can include it?

I recognise it's likely going to be a fairly trivial addon - but it's still an addon and my point remains - it would be good if several could be provided either as part of the core, or by way of supported addon.

 

[Mike]

The main thing would be loading these files, though you could likely put them virtually anywhere and include the necessary autoloader. The AWS SDK is a huge amount of overhead (in terms of size/files to include in the core download) for very few people who might potentially use it, for example. (We have done this with a couple libraries already but we could theoretically load a huge amount more to cover all sorts of edge cases.)

 

[Chris D]

I haven't actually tested it, but I don't think you'll need an Add-on, just a config:

$config['fsAdapters'] = [
    'data' => function () {
        return new \League\Flysystem\Filesystem(
            new \League\Flysystem\AwsS3v3\AwsS3Adapter(
                new \Aws\S3\S3Client([
                    'credentials' => [
                        'key'    => 'your-key',
                        'secret' => 'your-secret'
                    ],
                    'region' => 'your-region',
                    'version' => 'latest|version',
                ]),
                'your-bucket-name'
        ));
    },
];

and the adapter itself including it's dependencies.

Slightly more complete example (you need to override the data path too to ensure paths are prefixed with the bucket URL):

$config['fsAdapters'] = [
   'data' => function()
   {
      $client = new \Aws\S3\S3Client([
         'credentials' => [
            'key'    => 'ABC',
            'secret' => '123'
         ],
         'region' => 'eu-west-2',
         'version' => 'latest',
      ]);

      return new \League\Flysystem\AwsS3v3\AwsS3Adapter($client, 'a.a.com', 'data');
   }
];

$config['externalDataUrl'] = function($externalPath, $canonical)
{
   return 'https://s3.eu-west-2.amazonaws.com/a.a.com/data/' . $externalPath;
};

 

[Kirby]

How do you propose to get the adapter itself into a location where the XF core can include it?

Put it wherever you want and call it's autoloader
If this is what you mean with "requires an XF 2 Add-on" you're right.
But as Mike has already pointed out, it'd dont't think its a viable option to bundle a huge amount of dependencies that (this is my assumption) the majority of customers would never need.

 

[Chris D]

It's not an invalid suggestion. For example we could roll our own AWS client and Flysystem adapter which is much more lightweight and only covers the very specific endpoints we need to handle S3.

Bear in mind the AWS client that the official Flysystem S3 adapter uses covers every aspect of AWS which is humongous. It'd be nice actually if they componentised it a bit. Compressed it's 1.25MB. Uncompressed it's nearly 8MB.

But, as I said in the other thread and has been said here, advanced users can most likely roll it themselves, or an add-on might be able to simplify and roll the process up slightly (even if it's just a DIY tutorial) so something like that should be sufficient in the meantime.

 

[Jake B.]

Slightly more complete example (you need to override the data path too to ensure paths are prefixed with the bucket URL):

$config['fsAdapters'] = [
   'data' => function()
   {
      $client = new \Aws\S3\S3Client([
         'credentials' => [
            'key'    => 'ABC',
            'secret' => '123'
         ],
         'region' => 'eu-west-2',
         'version' => 'latest',
      ]);

      return new \League\Flysystem\AwsS3v3\AwsS3Adapter($client, 'a.a.com', 'data');
   }
];

$config['externalDataUrl'] = function($externalPath, $canonical)
{
   return 'https://s3.eu-west-2.amazonaws.com/a.a.com/data/' . $externalPath;
};

I assume something similar should work for internal_data as well?

 

[Alpha1]

I would love a DigitalOcean Spaces implementation: https://flysystem.thephpleague.com/docs/adapter/digitalocean-spaces/

The DO Spaces api are compatible with those of S3, from Flysystem’s perspective this means you can use the league/flysystem-aws-s3-v3 adapter.

 

[Jake B.]

I would love a DigitalOcean Spaces implementation: https://flysystem.thephpleague.com/docs/adapter/digitalocean-spaces/

From their documentation it seems like you'd just use the S3 handler with the code Chris provided above, just add in the endpoint parameter in their documentation

 

[Alpha1]

I'd like to see a trimmed down version which doesn't load a massive SDK.

 

[Jake B.]

it's not going to make much of a difference if you're installing it yourself locally, it still should only load what it needs if they included the full thing in XF it would just make the package everyone has to download ~13M larger

 

[lantek]

Does anyone have this working on AWS S3? Thanks

 

[abbeybas]

I would be very interested in thi as well!

 

[Don Daniello]

@Chris D The unfortunate part here is that attachments are stored in the internal-data:// mount where also other things are stored (imported_xml, install-lock.php, oembed_cache, image_cache, file_check, sitemaps/, etc), which are not as suitable for remote filesystem as attachments.

Architecturally, it would be more flexible if attachments were stored in a separate mount (aka filesystem) that could use a different underlying storage. The data:// mount is limited to thumbnails and avatars, which makes it easy to point to remote filesystem.

The attachment:// mount would be a private bucket (in case of most Cloud Object Storage solutions such as S3 or Swift) with access secured by temporary URLs (most flysystem adapters support generating these). This way, the action of downloading an attachment would be a redirect to a signed expiring URL, instead of streaming the attachment through the webserver.

 

[Xon]

Official tutorial with add-on; Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x for S3 compatible end-points