Using DigitalOcean Spaces or Amazon S3 for file storage

[dunowat]

From a REST protocol perspective, head simply gets the meta data of an object (size, properties, etc).
Get returns the stuff from head plus the object itself.

How the addon actually uses it, i'd need to research, but it's likely that head requests are done in a rebuild data call, or some other lookup ABOUT the image, but not actually bringing the image back.

It's also possible that your bucket security needs to be looked at.

Thanks, that's insightful. I'm not actually pulling downloads which is why I was wondering where those calls are coming from. It could be permissions.

The bucket is a single bucket set to either private or public. This serves both /data & /internal_data Is there a way for me to use for two different buckets one for /data (public) and one for /internal_data (private) ?



When I set the bucket to private, then I have issues with 'public-read' errors and nothing shows.

 

[briansol]

I use one bucket with subfolders

my private data isn't actually private. i don't really use attachments in the admin area and even if i did, someone would have to guess a filename. I left it open for caching/performance. I set up some headers with the https://xenforo.com/community/threads/image-attachment-cache-control.184359/ addon and also some cloudflare rules and also run an image optimizer plugin (TH, now dead) I outlined this over here:

C

Thread 'Moving UK host to UK shared or VPS - Suggestions ?'

Jul 6, 2022

We're currently based on a VPS in the UK. We've been given potential notice to leave because they want everyone using there own in-house management software rather than plesk or cpanel. While this isn't a major issue in itself, it does force our hand a bit because it's just so expensive anyway.

The forum has about 275,000 posts, 45,000 threads and 85,000 members. It has approximately 14,000+ page views a day.

Most of the resource (storage and bandwidth) is pictures which are currently on an external CDN. The CDN in the last 30 days served 692GB of data, 142GB of storage, and...

• cjwinternet
• Replies: 17
• Forum: Server configuration and hosting

from an aws perspective, i don't know why you're seeing a lot of requests if you aren't serving images... could be lots of things.
there's not a lot of visibility into what hits your bucket.

you need to set the ACL and IAM properly. I struggled with this setup the first time around.

basic outline:

Create an IAM user
and create a permission policy for s3 access to attach to that user.

mine looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:putObject",
                "s3:putObjectAcl",
                "s3:ReplicateObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucketname",
                "arn:aws:s3:::mybucketname/*"
            ]
        }
    ]
}

attach that user to yourbucketname via the access keys (and that goes in the xf config file)


then set the bucket up like this:
see:

Thread 'Using DigitalOcean Spaces or Amazon S3 for file storage'

Nov 15, 2018

Chris D submitted a new resource:

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x - The same concepts can be applied to other adapters too.

Why this guide?

Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file system as it adds a layer of abstraction between the code and a file system. It means that it provides a consistent API for performing file system operations so that whether the file system is a local disk-based file system or a...

• Chris D
• Replies: 815
• Forum: Tips and guides [2.x]

• 
• 

for the bucket permissions


Hope this helps a little bit

 

[dunowat]

from an aws perspective, i don't know why you're seeing a lot of requests if you aren't serving images... could be lots of things.
there's not a lot of visibility into what hits your bucket.

If images are served then these are requests right? I think the issue was I was requesting full size images. It seems to have slowed down now that I've switch layouts to use thumbnails.

The setup actually works... I'm just trying ot understand the number of calls i'm getting. I had some ~25k calls for about 1k users, which would make sense if that gets pulled with the full size image.

I'm just trying to get a better understanding of the add on. It works well but I do have that one problem with public/private calls.

Thanks for the information. It's very helpful to understand how others use it too.

 

[Paul B]

I asked this in the module forum but perhaps I should ask in the customer support forum since this is a xenforo addon.

Questions related to the resource should be posted in the associated thread, regardless who created the resource.

I have merged the threads.

 

[dunowat]

Questions related to the resource should be posted in the associated thread, regardless who created the resource.

I have merged the threads.

With all due respect. I posted a few days ago here without hearing anything. So I asked in the customer forum

 

[Paul B]

Regardless, the rules for all resources are the same.

 

[RandallC]

Configured through the Config and kept getting errors. Gave up

 

[dunowat]

Configured through the Config and kept getting errors. Gave up

what's your error bro? It's worth it to setup..

 

[RandallC]

I do the config in S3 and IAM and get the key under users and Xenforo gives me the error.

Basically something lile this : https://xenforo.com/community/threa...storage-in-xf-2-1.156282/page-33#post-1609024

 

[dunowat]

I do the config in S3 and IAM and get the key under users and Xenforo gives me the error.

Basically something lile this : https://xenforo.com/community/threa...storage-in-xf-2-1.156282/page-33#post-1609024

400 issue is your permissions, so you should reset and start that over.

If you want to go on an easier path with permissions to test it out use backblaze.com Their buckets have limited capabilities, plus you can integrate with Cloudflare for free. I'm running this setup with really good results.

Set the api and api key, setup DNS and the paths.. You don't need to setup Cloudflare either. You can just do backblaze and it'll be good.

 

[RandallC]

I like to use Amazon s3, but for some reason when I follow the guide I get error 400 which has been the permissions even though I create a bucket, add user, add permissions via JSON. Link the user to the bucket create the keys. make the config.php edits.

 

[RandallC]

Got everything set and running now, Extremely happy about stuff. Just one small Issue, Avatars are not all showing up on the pages even though when you go to the persons profile(still not showing) but when you click on it it opens.

 

[JonathanC]

I'm currently using Digital Ocean as my CDN, currently have no errors and items from XFMG are being uploaded to the CDN, and streaming from the CDN. The issue is, it will upload attachments, but Xenforo is serving the images not the CDN. Is anyone else having this issue? I'm wondering if it may be a compatibility issue with version 2.3


Edit: I also tried uninstalling the "add-on" since 2.3 now has native support but it completely breaks the images, and I'm unable to upload anything.

 

[digitalpoint]

That’s normal behavior for any abstracted filesystem assigned to the internal_data mount point. Specifically it’s works that way because the ability to view attachments are controlled by per-user permissions in XenForo (potentially some users are able to view them, while others aren’t). You can’t do the permission checks at the CDN edge, so the application (XenForo) does the check and then passes through the attachment if the user is allowed to view it.

There are ways around it (presigned URLs, force caching of attachment URL, etc), but I only know the details of how to do that with Cloudflare, not Digital Ocean.

TL;DR: it’s expected behavior because XenForo acts as the gatekeeper for which users can view each attachment.

 

[Jason Trumpio]

FYI it appears there are some compatibility issues with 2.3.

• ErrorException: Fatal Error: Declaration of Aws\S3\ObjectUploader::promise() must be compatible with GuzzleHttp\Promise\PromisorInterface::promise(): GuzzleHttp\Promise\PromiseInterface src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/ObjectUploader.php:63
  Generated by: Jason Jul 5, 2024 at 10:12 AM

 

[Chris D]

You can uninstall this if you are using XF 2.3.

 

[Jason Trumpio]

You can uninstall this if you are using XF 2.3.

Has S3 object storage been baked in and I missed it? Currently the way I have this architected S3 is considerably cheaper than EFS for autoscaling and media storage.

 

[Chris D]

XF2 has always had support for object storage. All this resource has ever done is add the AWS SDK. It was never included in XF because it is unreasonably huge.

Starting with 2.3 we have included a subset of the SDK therefore this “add-on” is no longer needed.

Uninstall it and everything you’ve set up will continue working as normal.

 

[BrettC]

XF2 has always had support for object storage. All this resource has ever done is add the AWS SDK. It was never included in XF because it is unreasonably huge.

Starting with 2.3 we have included a subset of the SDK therefore this “add-on” is no longer needed.

Uninstall it and everything you’ve set up will continue working as normal.

Hi Chris, unless I've missed it somewhere or searched for the wrong terms... are there any documents/tutorials posted for this 2.3 feature making this addon no longer needed?

 

[Chris D]

It was mentioned in one of the have you seen posts.

No documentation needed. To set it up you still need to follow the instructions in the resource guide, you just don’t need the add-on.

If it is already set up and working, you can just uninstall the add-on.