Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Overview Updates (4) Reviews (7) Discussion
mcatze said:
Is it possible to use https://aws.amazon.com/mediaconvert/ with AWS S3 and XenForo Media Gallery for Videos?
Click to expand...
@mcatze did you find out if this can be used to transcode videos with Media Gallery? I think an option to use a processing service instead of FFMPEG on a VPS with limited resources would be a great option. Using a vimeo account or similar to handle processing would create a more robust option for videos than stretching CPU resources when videos are processed. Vimeo also serves the video with a CDN so less traffic on site bandwidth is beneficial.
 
getting the following server errors, any idea what might be wrong?

Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://abcdef.eu-central-1.linodeo...3/13342-57c40eba101384e95a9dc75182d3899b.data"; AWS HTTP error: Server error: HEAD https://abcdef.eu-central-1.linodeobjects.com/internal_data/attachments/13/13342-57c40eba101384e95a9dc75182d3899b.data resulted in a 502 Bad Gateway response (server): 502 Bad Gateway - src/addons/XFAws/_vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:191
 
@Chris D, thank you for this add-on and excellent instructions.

I just ported the JetBoaters.net forum's 150 GB of attachments to DO Spaces and found your instructions to be spot on.

One point that I wondered about is would it be possible and beneficial to add an internal_dataUrl entry similar to the externalDataUrl entry to offload serving attachments from the forum server?
 
There is no internalDataPath and due to various approaches - mainly permission checks - we have to serve the attachments from a URL where we can do those checks.

The offloading issue is indeed an issue that we're conscious of but we've not take steps to address that yet.
 
BruceW said:
@Chris D, thank you for this add-on and excellent instructions.

I just ported the JetBoaters.net forum's 150 GB of attachments to DO Spaces and found your instructions to be spot on.

One point that I wondered about is would it be possible and beneficial to add an internal_dataUrl entry similar to the externalDataUrl entry to offload serving attachments from the forum server?
Click to expand...
What I do is to make CloudFlare cache the /attachments route for one day. You're going to face with two problems:

1) attachments protected by the permission system will be accessible to anyone
2) If you have very big attachments, it might go against cloudflare policies to store those attachment in their edge servers.

In my particular case none of those problems affected me that much so I decided to go with that route and it's been working perfectly fine.
 
Thanks @Chris D.

If I may ask a follow up question, is the S3 based internal_data cached locally on the server?

I know that when I have used s3fs, which I find to be terribly inefficient, that it creates a large local cache.
 
from my experience, even sitemap is generated remotely. realized it when i saw backblaze storing multiple versions of the sitemap file while it was being generated.
 
I'm experiencing strange behavior with XFMG whereby video upload works but playback fails due to 403

HTTP load failed with status 403. Load of media resource

Other video attachments in normal forum posts play well using the default player but XFMG videos fail to play, it seems like something to do with access being denied, is the XFMG video player different from the one used in other parts of the forum?

The media could not be loaded, either because the server or network failed or because the format is not supported.
Click to expand...

@Chris D any idea what could be wrong?
 
Just updated my forum to use Digitalocean Spaces. It works great and we easy to update. The biggest pain point was waiting on the 12 hours of file transfers via ftp.

We ran out of storage on our Bluehost VPS so this was a lifesaver.

Would it be worth exploring the Digitalocean's CDN option or leaving the set up as is? What is the best advice here.

@Chris D With xenForo 2.2 posts over the last week, it looks like 2.2 will be here before too long. Do we expect this addon to work with 2.2 at launch? Thanks
 
RenegadeMonster said:
Just updated my forum to use Digitalocean Spaces. It works great and we easy to update. The biggest pain point was waiting on the 12 hours of file transfers via ftp.

We ran out of storage on our Bluehost VPS so this was a lifesaver.

Would it be worth exploring the Digitalocean's CDN option or leaving the set up as is? What is the best advice here.

@Chris D With xenForo 2.2 posts over the last week, it looks like 2.2 will be here before too long. Do we expect this addon to work with 2.2 at launch? Thanks
Click to expand...

I use the cdn option with custom domain. It works great because you can setup your own domain instead of the digitalocean's one and in theory the files should be served from a closer location to your visitors.
 
JulianD said:
I use the cdn option with custom domain. It works great because you can setup your own domain instead of the digitalocean's one and in theory the files should be served from a closer location to your visitors.
Click to expand...

Thanks for the info.

I just enabled the CDN and will wait a couple days for it to propagate before changing the endpoints.
I currently have the CDN set to update every hour. Is this the best setting for xenForo? If not what is?

Also, are there ever any issue where band new post with uploaded images are broken until they propagate? And when you said you use a custom domain. Are you using the same domain your forum is on? Or a different domain?
 
ahem. amazon does have a dedicated database component (https://aws.amazon.com/rds/mysql/) as part of aws. i used it for a while. it gets pretty expensive. and does not really give you any added advantage because every database query is being fetched from an external data center (assuming you are not using aws for hosting).

object storage is designed for static objects. database should be in the same datacenter as the actual site content for best performance.

update: here is the digital ocean product as mentioned by chris: https://www.digitalocean.com/products/managed-databases-mysql/
 
DigitalOcean does MySQL database hosting too FWIW but, yeah, it's totally separate to this. This is exclusively for file storage.
 
OperaManiac said:
whoa. how many days are we talking about here! i have only been able to use the cdn endpoint for the data folder. nothing else works here 'endpoint' => 'https://xxxx.digitaloceanspaces.com'!
Click to expand...

I just ran into the same thing. The cdn in the endpoint causes an error when uploading.

Did you ever find a solution? I too currently am using the endpoint on the data folder after turning it on because of this.

Also, does using the cdn here also use it for the internal_data folder?
 
i have moved on to backblaze since then. the only thing that worked for me was using a custom domain by uploading ssl details at their end in the cdn section. that applies to external path. from my understanding you do not need to use the cdn url in primary config because it is primarily used for uploads and attachments. both of which are handled by xenforo which means cdn has no benefit here because server and object storage is likely to be in the same data center for most users?
 
You must log in or register to reply here.
Top Bottom