Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

[dunowat]

From a REST protocol perspective, head simply gets the meta data of an object (size, properties, etc).
Get returns the stuff from head plus the object itself.

How the addon actually uses it, i'd need to research, but it's likely that head requests are done in a rebuild data call, or some other lookup ABOUT the image, but not actually bringing the image back.

It's also possible that your bucket security needs to be looked at.

Thanks, that's insightful. I'm not actually pulling downloads which is why I was wondering where those calls are coming from. It could be permissions.

The bucket is a single bucket set to either private or public. This serves both /data & /internal_data Is there a way for me to use for two different buckets one for /data (public) and one for /internal_data (private) ?



When I set the bucket to private, then I have issues with 'public-read' errors and nothing shows.

 

[briansol]

I use one bucket with subfolders

my private data isn't actually private. i don't really use attachments in the admin area and even if i did, someone would have to guess a filename. I left it open for caching/performance. I set up some headers with the https://xenforo.com/community/threads/image-attachment-cache-control.184359/ addon and also some cloudflare rules and also run an image optimizer plugin (TH, now dead) I outlined this over here:

C

Thread 'Moving UK host to UK shared or VPS - Suggestions ?'

Jul 6, 2022

We're currently based on a VPS in the UK. We've been given potential notice to leave because they want everyone using there own in-house management software rather than plesk or cpanel. While this isn't a major issue in itself, it does force our hand a bit because it's just so expensive anyway.

The forum has about 275,000 posts, 45,000 threads and 85,000 members. It has approximately 14,000+ page views a day.

Most of the resource (storage and bandwidth) is pictures which are currently on an external CDN. The CDN in the last 30 days served 692GB of data, 142GB of storage, and...

• cjwinternet
• Replies: 17
• Forum: Server configuration and hosting

from an aws perspective, i don't know why you're seeing a lot of requests if you aren't serving images... could be lots of things.
there's not a lot of visibility into what hits your bucket.

you need to set the ACL and IAM properly. I struggled with this setup the first time around.

basic outline:

Create an IAM user
and create a permission policy for s3 access to attach to that user.

mine looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:putObject",
                "s3:putObjectAcl",
                "s3:ReplicateObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucketname",
                "arn:aws:s3:::mybucketname/*"
            ]
        }
    ]
}

attach that user to yourbucketname via the access keys (and that goes in the xf config file)


then set the bucket up like this:
see:

Thread 'Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+'

Nov 15, 2018

Chris D submitted a new resource:

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x - The same concepts can be applied to other adapters too.

Why this guide?

Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file system as it adds a layer of abstraction between the code and a file system. It means that it provides a consistent API for performing file system operations so that whether the file system is a local disk-based file system or a...

• Chris D
• Replies: 705
• Forum: Tips and guides [2.x]

• 
• 

for the bucket permissions


Hope this helps a little bit

 

[dunowat]

from an aws perspective, i don't know why you're seeing a lot of requests if you aren't serving images... could be lots of things.
there's not a lot of visibility into what hits your bucket.

If images are served then these are requests right? I think the issue was I was requesting full size images. It seems to have slowed down now that I've switch layouts to use thumbnails.

The setup actually works... I'm just trying ot understand the number of calls i'm getting. I had some ~25k calls for about 1k users, which would make sense if that gets pulled with the full size image.

I'm just trying to get a better understanding of the add on. It works well but I do have that one problem with public/private calls.

Thanks for the information. It's very helpful to understand how others use it too.

 

[Paul B]

I asked this in the module forum but perhaps I should ask in the customer support forum since this is a xenforo addon.

Questions related to the resource should be posted in the associated thread, regardless who created the resource.

I have merged the threads.

 

[dunowat]

Questions related to the resource should be posted in the associated thread, regardless who created the resource.

I have merged the threads.

With all due respect. I posted a few days ago here without hearing anything. So I asked in the customer forum

 

[Paul B]

Regardless, the rules for all resources are the same.