Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Code:
Traceback (most recent call last):
  File "/bin/s3cmd", line 3121, in <module>
    rc = main()
  File "/bin/s3cmd", line 3030, in main
    rc = cmd_func(args)
  File "/bin/s3cmd", line 375, in cmd_object_put
    destination_base_uri = S3Uri(args.pop())
  File "/usr/lib/python2.7/site-packages/S3/S3Uri.py", line 42, in __new__
    raise ValueError("%s: not a recognized URI" % string)
ValueError: https://bucketname.s3.us-east-1.amazonaws.com/data/: not a recognized URI
 
and it's in your bucket? that might be someone elses bucket.


you used us-east when you showed eu-west to me previously. make sure it's in the right region.
 
Bit late to the party on this one, however interesting read (especially as, whilst i know of AWS S3 storage, I have never been seen it "in use".

Does anyone have any examples (even approx ones) of the potential cost savings going the S3 route? Most of my bandwidth is pictures (around 40Gb hosted currently - 350Gb/Month transferred monthly).... I tried going through the S3 Calculator, however it confused the life out of me :)

Any suggestions and pointers/advice appreciated!
Paul
 
all attach in normal forum but i get this error with XFMG
  • League\Flysystem\FileNotFoundException: File not found at path: xfmg/original/2/2433-46a35544634236b962f1e4bbbf180e83.data
  • src/vendor/league/flysystem/src/Filesystem.php:389

Thanks
 
Bit late to the party on this one, however interesting read (especially as, whilst i know of AWS S3 storage, I have never been seen it "in use".

Does anyone have any examples (even approx ones) of the potential cost savings going the S3 route? Most of my bandwidth is pictures (around 40Gb hosted currently - 350Gb/Month transferred monthly).... I tried going through the S3 Calculator, however it confused the life out of me :)

Any suggestions and pointers/advice appreciated!
Paul

I would be interested in knowing this as well, like you it made my head hurt trying to work out the S3 calculator properly
 
It's hard to calculate.

you pay for both storage and ingress/egress.

if you have cloudflare or cloudfront in front of the bucket, you have caching to consider as well.

For me, it wasn't about cost savings directly.
moving the attachments off my VPS allowed me to downgrade my VPS to a cheaper unit (less disk).
It also made backups FAR quicker (min instead of hours) and reduced the size of the backup so if i ever needed it, i could restore quickly. I don't back up s3. I trust s3. Once a quarter or so i'll copy s3 buckets to another AZ and throw it in glacier cold storage for cheap backup.

So, i went from a $50 vps to a 25 vps, and added about $8 in aws charges. Net, i'm reduced cost, faster backups, and better delivery/scale from cdn.

I have about 50 gb of images (highly optimized) and serve about 10gb a day.
 
It's hard to calculate.

you pay for both storage and ingress/egress.

if you have cloudflare or cloudfront in front of the bucket, you have caching to consider as well.

For me, it wasn't about cost savings directly.
moving the attachments off my VPS allowed me to downgrade my VPS to a cheaper unit (less disk).
It also made backups FAR quicker (min instead of hours) and reduced the size of the backup so if i ever needed it, i could restore quickly. I don't back up s3. I trust s3. Once a quarter or so i'll copy s3 buckets to another AZ and throw it in glacier cold storage for cheap backup.

So, i went from a $50 vps to a 25 vps, and added about $8 in aws charges. Net, i'm reduced cost, faster backups, and better delivery/scale from cdn.

I have about 50 gb of images (highly optimized) and serve about 10gb a day.

Many thanks for that, it is very helpful.

We are just starting up so would like to be aware of all options/costs sooner rather than later

Planning ahead as the saying goes :)
 
I have a forum where I configured S3 storage like this:

https://xenforo.com/community/resou...or-amazon-s3-for-file-storage-in-xf-2-x.6805/

PHP:
$config['fsAdapters']['data'] = function() use($s3)

{

return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'cdn.**.net', '/forum/');

};

$config['fsAdapters']['internal-data'] = function() use($s3)

{

return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'cdn.***.net', '/forum/');

};

$config['externalDataUrl'] = function($externalPath, $canonical)

{

return '//cdn.***[B][I].net/[/I][/B]/' . $externalPath;

};

I manually moved all content to S3 with public permissions, and if I upload a new attachment to a discussion now it works. But I'm having a problem. All old attachments are not visible.

Esempio:

Immattagine.webp
As you can see, the first attachment that is not visible is the old one, while the second is what I added by doing:
Edit Message -> Add Thumbnail

Why are all the images not visible? a tool for reconstruction?
 
As I said, I uploaded the files to S3 and set the permissions, in fact if I edit the post and put the thumbnails it works correctly.

But the old link, before I set up the S3, the link stays like this, without S3:

HTML:
<div class="bbImageWrapper  js-lbImage" title="2nc0vuh.jpg" data-src="https://www.**.com/forum/attachments/2nc0vuh-jpg.23006/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1">
        <img src="https://www.**.com/forum/attachments/2nc0vuh-jpg.23006/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="2nc0vuh.jpg" title="2nc0vuh.jpg" width="300" height="300" loading="lazy">
    </div>

if I edit the message and select the attachment, he puts it with:

HTML:
<a href="https://www.**.com/forum/attachments/psu-ps4-jpg.9845/" target="_blank" class="js-lbImage" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-fancybox="lb-thread-82679" data-caption="<h4>Psu ps4.jpg</h4><p><a href=&quot;https:&amp;#x2F;&amp;#x2F;www.**.com&amp;#x2F;forum&amp;#x2F;threads&amp;#x2F;ps4-si-accende-e-subito-si-spegne.82679&amp;#x2F;#post-837213&quot; class=&quot;js-lightboxCloser&quot;>Brek · 17 Febbraio 2016 alle 20:44</a></p>" style="cursor: pointer;"><img src="//s3.**.net/**/attachments/6/6888-55e9062232a5d5f04ce164f5c1e36d0f.jpg" class="bbImage " style="" alt="Psu ps4.jpg" title="Psu ps4.jpg" width="256" height="160" loading="lazy"></a>

As you can see, the attachment in the PRE-S3 message has not been changed...
 
As I said, I uploaded the files to S3 and set the permissions, in fact if I edit the post and put the thumbnails it works correctly.
In your example above you're using "s3..." URLs for your attachments but in your posted config.php changes you're using "cdn..." URLs.

Are you actually using a CDN? If not, your config.php file should use the "s3..." URLs.
 
Top Bottom