Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Overview Updates (4) Reviews (7) Discussion
i haven't read anything about keeping copies of the attachment data on multiple platforms. or distributing load between them. you can probably set up a cron job to use something like rclone to mirror the relevant folders between multiple platforms. backblaze has an option to take a snapshot of your b2 bucket for instance plus there is versioning.
 
Do you know if this add on can make multiple s3-compatible calls for redundancy? I.e., use S3, DO, and Vultr at the same time.
 
this addon does not really do anything. it is just a connector to s3 api. so xenforo treats s3 as storage space for attachments and other uploads. any new upload would store data on the configured storage space. this cannot manage your multiple s3 locations. you would have to migrate/replicate data on your own. if you start using this addon, you have to copy your data on existing hosting server to s3 compatible storage space like vultr or s3 or do spaces yourself.

if you want redundancy, it would require custom code. you would have to run some code to mirror the three storage spaces. and then find a way to change the config.php file where you configure the s3 compatible storage space you want to use.
 
  • Like
Reactions: frm
How do people backup their data? I want to make back-up, even from "cloud" data, and want to have a consitent xenforo backup.

And when I decide to stop with Amazon S3, I can simply reupload all data to the attachments directory?
 
Check if the s3 compatible service you are using provides an option back up the bucket. B2 does by letting me create a snapshot and copy it to a dedicated snapshot bucket. Or you can use rclone type app to replicate your bucket on same or another service.

To remove s3 storage space, just copy back the entire data back and remove relevant lines from config.php file!
 
I don't back up s3. I assume it is stable and replicated. That may be naive and stupid but considering a store my cpanel backup on s3 too, if aws goes down on the same day as my web host, i'm probably heading for the hills any way...

I use vultr and am in NJ and have looked into their data tools but i stuck with s3. Frankly, i trust s3 more. if NJ gets hit with a nuke, it's all gone. s3 at least has a chance since it's globally dispersed.
 
Has anyone done any testing with 2.2 yet and this addon? Curious if we will be good to go when up 2.2 is released or if we should wait for this addon to be updated because of any bugs related to running this on 2.2.
 
having issues with this on 2.2


Fatal error: Uncaught Error: Class 'League\Flysystem\AwsS3v3\AwsS3Adapter' not found in /home/xxxx/public_html/src/config.php:41 Stack trace: #0 [internal function]: XF\App->{closure}() #1 /home/xxxxx/public_html/src/XF/FsMounts.php(17): call_user_func(Object(Closure)) #2 /home/xxxxxx/public_html/src/XF/App.php(1071): XF\FsMounts::loadDefaultMounts(Array) #3 /home/xxxxxx/public_html/src/XF/Container.php(28): XF\App->XF\{closure}(Object(XF\Container)) #4 /home/xxxxxx/public_html/src/XF/App.php(2540): XF\Container->offsetGet('fs') #5 /home/xxxxx/public_html/src/XF.php(865): XF\App->fs() #6 /home/xxxxx/public_html/src/XF/Util/File.php(738): XF::fs() #7 /home/xxxxxx/public_html/src/XF/Error.php(100): XF\Util\File::installLockExists() #8 /home/xxxxx/public_html/src/XF/App.php(2311): XF\Error->logException(Object(Error), true, '') #9 /home/xxxxx/public_html/src/XF.php(172): XF\App->logException(Object(Error), true) #10 [internal function]: XF::handleException(Object(Error)) #11 {main} thrown in /home/xxxxx/public_html/src/config.php on line 41
 
Nevermind - for some reason when creating the iam it didnt take the permissions from the json but I recreated them and now everything works.
 
Last edited:
Setting Up Amazon S3:

Accept any further default options until the bucket is created.
Click to expand...

I did that but the default settings are to make the entire bucket private with no endpoint. Im able to access my bucket fine using a program that can open S3 buckets, but Im getting a 403 error when trying to login to admin control panel in XF.

Code: 
[B]Aws\S3\Exception\S3Exception[/B]: Error executing "PutObject" on "https://xftest.s3.us-east-1.amazonaws.com/internal_data/install-lock.php"; AWS HTTP error: Client error: `PUT https://xftest.s3.us-east-1.amazonaws.com/internal_data/install-lock.php` resulted in a `403 Forbidden` response

The
https://xftest.s3.eu-west-2.amazonaws.com/data/
Click to expand...
example config is beginning to make me think it implies that an endpoint must be created for XF. My program that lets me view my buckets only needed s3.amazonaws.com and the ID and secret key to get access to the bucket.
 
I was getting the same error for put object, on my IAM user I had to redo the permission- when I setup the IAM it didnt work but after the user is created it now works when I went to the IAM user and permissions and submited to JSON again. Now everything works as it should
 
briansol said:
xenforo.com

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Tested this out with DigitalOcean and it works great! The only point of feedback would be to add support for DigitalOcean's Spaces CDN endpoint. CDN comes for free when you use DigitalOcean Spaces, but when you try to use it with this system it gives a 403 error...
xenforo.com xenforo.com
Click to expand...

Thanks Brian, to my understanding, XF is using the REST API to access S3 and in aws the only times the REST API is accessible is when using a Origin Access Identity with CloudFront or when "block public access" is not enabled. This article speaks about cloudfront + 403 errors however it may apply here. https://aws.amazon.com/premiumsupport/knowledge-center/s3-rest-api-cloudfront-error-403/
I believe using the bucket with "block public access" would only work if XF accessed it using the aws sdk or cli.
 
blablafixer said:
Thanks Brian, to my understanding, XF is using the REST API to access S3 and in aws the only times the REST API is accessible is when using a Origin Access Identity with CloudFront or when "block public access" is not enabled. This article speaks about cloudfront + 403 errors however it may apply here. https://aws.amazon.com/premiumsupport/knowledge-center/s3-rest-api-cloudfront-error-403/
I believe using the bucket with "block public access" would only work if XF accessed it using the aws sdk or cli.
Click to expand...
My settings are like the one in the screen print. the bottom two matched.
 
@Chris D

It would be good to note in the guide for aws s3 that the "Block public access settings" need to look like this:


Block all public access
Off
  • Block public access to buckets and objects granted through new access control lists (ACLs)
    Off

  • Block public access to buckets and objects granted through any access control lists (ACLs)
    Off

  • Block public access to buckets and objects granted through new public bucket or access point policies
    On

  • Block public and cross-account access to buckets and objects through any public bucket or access point policies
    On

access.png

Because by default the bucket now gets created with Block all public access: On. Which doesn't allow the ACL "public read" permission on objects to function.

This is at step number
6. Accept any further default options until the bucket is created.
Click to expand...



Also later down the line, can we get an option to offload addon files to s3 as well? They are the only non-core persisted files I can think of that are remaining.
... Then if we want to get super advanced XF could have an option to be deployable as a docker container, just require the DB, S3 bucket location(s), and license info in a config file. Server deployment would be very flexible.
I can make a suggestion if you think that's a good idea.
 
  • Like
Reactions: fly
In case this helps someone - the

$config['externalDataUrl'] = function($externalPath, $canonical)
Click to expand...

part of the config is for specifying the URL the viewers of your forum will be given for the location for your S3 data assets. You can specify a CDN here to optimize delivery of your data assets. (:
 
  • Like
Reactions: fly
Another improvement would be editing this:


The following guide will work in either XF 2.0 or XF 2.1. You need to ensure you are using the correct version of the add-on files.

Both are included when you click the download button, please ensure you download version 2.0.x if you are using XenForo 2.0.x and please ensure you download version 2.1.x if you are using XenForo 2.1.x.
Click to expand...

to


The following guide will work in either XF 2.0, XF 2.1, or XF 2.2. You need to ensure you are using the correct version of the add-on files.

Two versions are included when you click the download button, please ensure you download version 2.0.x if you are using XenForo 2.0.x and please ensure you download version 2.1.x if you are using XenForo 2.1.x or XenForo 2.2.x.
 
You must log in or register to reply here.

Similar threads

B
Using Digital Ocean Spaces or Amazon S3 for internal_data / attachments
Replies
3
Views
1K
Vlady
V
Top Bottom