Using DigitalOcean Spaces or Amazon S3 for file storage

Using DigitalOcean Spaces or Amazon S3 for file storage

No permission to download
Overview Updates (6) Reviews (7) Discussion
Yeah what threw me off was when inspecting images, some had my site's URL and some had amazon's s3 URL. I see that was the same before and the one's exposing the amazon URL have the [ATTACH]xxxxxx[/ATTACH] code.
 
Last edited:
Wow, okay. So this brings up a question for me....

What happens when I backup the forum to test out new features? If I upload an attachment to test-forum, will it mess with attachments for the main forum?
 
In the same way that your test installation would point to a cloned database, you would want to point it at cloned buckets as well, or remove the configuration and ignore the missing files.
 
Yikes. Okay. I did not do that. In fact, I uploaded about 50 new attachments through the test-forum yesterday. I see them in the bucket, because they all have yesterday's date. I plan on adding those same attachments to the main-forum. How would you suggest I proceed at this stage?
 
You can remove them from the bucket if you’re confident they’re the right ones. Or you can leave them and it’s probably fine other than them taking up space. The bigger deal would be modifying or deleting attachments on the test installation.
 
Jeremy P said:
You can remove them from the bucket if you’re confident they’re the right ones. Or you can leave them and it’s probably fine other than them taking up space. The bigger deal would be modifying or deleting attachments on the test installation.
Click to expand...

This ☝️ is good to know....

Thanks for your help, and I'm probably going down a rabbit-hole with this, but I'm a bit fixated on making it work and doing it correctly.

In my Amazon S3 I create another bucket called test-forum. I then copied all the files from my forum bucket to my test-forum bucket.

I also changed my config.php anywhere it said 'forum', i changed it to 'test-forum'.

Is this the correct way of making a clone, because I'm not seeing any images on my test-forum any more....

Thanks,
 
Last edited:
It depends on the provider, and it's been a while since I've touched the AWS console to be honest with you, but it may have to do with permissions in IAM. The instructions in the original post show you how to set up credentials for a given set of buckets (resources).
 
I am getting this error every time I try and access an attachment.

Code: 
An exception occurred: [InvalidArgumentException] Must pass valid resource in in src/XF/Http/ResponseStream.php on line 18

[LIST=1]
[*]XF\Http\ResponseStream->__construct() in src/XF/Http/Response.php at line 262
[*]XF\Http\Response->responseStream() in src/XF/Pub/View/Attachment/View.php at line 28
[*]XF\Pub\View\Attachment\View->renderRaw() in src/XF/Mvc/Renderer/AbstractRenderer.php at line 92
[*]XF\Mvc\Renderer\AbstractRenderer->renderViewObject() in src/XF/Mvc/Renderer/Raw.php at line 39
[*]XF\Mvc\Renderer\Raw->renderView() in src/XF/Mvc/Dispatcher.php at line 471
[*]XF\Mvc\Dispatcher->renderView() in src/XF/Mvc/Dispatcher.php at line 453
[*]XF\Mvc\Dispatcher->renderReply() in src/XF/Mvc/Dispatcher.php at line 412
[*]XF\Mvc\Dispatcher->render() in src/XF/Mvc/Dispatcher.php at line 66
[*]XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2826
[*]XF\App->run() in src/XF.php at line 806
[*]XF::runApp() in index.php at line 23
[/LIST]

and this shows on the admin page

Code: 
[HEADING=2]Stack trace[/HEADING]
#0 src/XF/Http/Response.php(262): XF\Http\ResponseStream->__construct(false, 459720)
#1 src/XF/Pub/View/Attachment/View.php(28): XF\Http\Response->responseStream(false, 459720)
#2 src/XF/Mvc/Renderer/AbstractRenderer.php(92): XF\Pub\View\Attachment\View->renderRaw()
#3 src/XF/Mvc/Renderer/Raw.php(39): XF\Mvc\Renderer\AbstractRenderer->renderViewObject('XF:Attachment\\V...', '', Array)
#4 src/XF/Mvc/Dispatcher.php(471): XF\Mvc\Renderer\Raw->renderView('XF:Attachment\\V...', '', Array)
#5 src/XF/Mvc/Dispatcher.php(453): XF\Mvc\Dispatcher->renderView(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#6 src/XF/Mvc/Dispatcher.php(412): XF\Mvc\Dispatcher->renderReply(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#7 src/XF/Mvc/Dispatcher.php(66): XF\Mvc\Dispatcher->render(Object(XF\Mvc\Reply\View), 'raw')
#8 src/XF/App.php(2826): XF\Mvc\Dispatcher->run()
#9 src/XF.php(806): XF\App->run()
#10 index.php(23): XF::runApp('XF\\Pub\\App')
#11 {main}

I had Redis caching enabled but disabled it and still get it. I am on XF 2.3.4 with PHP 8.3.14
WLgTr8LrKH.png


My config is this, secrets redacted,
Code: 
$s3 = function()
{
   return new \Aws\S3\S3Client([
      'credentials' => [
         'key' => '',
         'secret' => ''
      ],
      'region' => 'us-east-1',
      'version' => 'latest',
      'endpoint' => 'https://s3.us-east-1.amazonaws.com'
   ]);
};

$config['fsAdapters']['data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'cdn.totalfreedom.tf', 'data');
};

$config['externalDataUrl'] = function($externalPath, $canonical)
{
   return 'https://cdn.totalfreedom.tf/data/' . $externalPath;
};

$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'cdn.totalfreedom.tf', 'internal_data');
};

Also note that cdn.totalfreedom.tf is a CNAME record to s3.us-east-1.amazonaws.com and the actual bucket name is cdn.totalfreedom.tf

Edit: I disabled Attachment Improvements by Xon and I still get the error.
 
Last edited:
So then tell me why the v2.16 update does not work with the latest plugin application? I suspect that it will be left behind in favor of V2.3 development. Many of us have no intention of upgrading any time soon. Our platforms are solid and secure, and function well. I have to deal with this, as developers such as Chris and Jeremy continue to tell us sort of what were going to have-without regard to what we really need.

I know that sounds harsh and might even aggravate some people. The lack of response to my queries and sorting them out tells me the tale--the same one I have heard for over 20 years with a variety of form software's.

And the extension still does not work right for me in its most updated version...
 
We're happy to provide support but it will likely require much more clarity than what your previous message was in this thread. To be clear, your last message was more of a vague journey than a declaration of the specific issues you were experiencing. My interpretation of it by the end, at least, was you were "back to running the baseline" which you had said was working perfectly. It also seemed you had been facing some other issues, some of which didn't seem entirely relevant.

For example, the following is not really helpful from a troubleshooting perspective:

It did not like that a bit and choked with a bunch of red error message lines. Something having to do with the location of flags for a Snog addon.
Click to expand...

I think ultimately it would be best to start from scratch, probably in a test environment, and we can help you with each issue that comes up separately as long as we're given the specific steps you took and what the specific errors were at that point.
 
Sorry Chris, but I thought I posted enough detail that this could be sussed out. Perhaps my mistake, but I was ghosted when detailing the last details.

I would dearly like to use the latest S3 extension, but it simply does not work for me. The previous one does. Life will go on without incessant updates. I know this because I have been doing this for 25 years across a number of forum platforms.

My site is a live production platform. It is far beyond Laragon or WAMP test beds.

The only thing that failed is the S3 update. Really?

The need has passed. Onward and upward. Thank you for your service.
 
Just starting using S3 buckets for data and internal_data with Xenforo.

I had to enable ACLs which is not recommended anymore, but more importantly, I found out that it is trying to enable public access for files in internal_data, which sounds counter intuitive at least to me:


Error executing "PutObject" on "https://****.s3.eu-west-3.amazonaws.com/internal_data/oembed_cache/1/1064-****.json"; AWS HTTP error: Client error: PUT https://****.s3.eu-west-3.amazonaws.com/internal_data/oembed_cache/1/1064-****c.json resulted in a 403 Forbidden response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>User: arn:aws:iam:::user/xen (truncated...) AccessDenied (client): User: arn:aws:iam:::user/xen** is not authorized to perform: s3: putObject on resource: "arn:aws:s3:::/internal_data/oembed_cache/1/1064-.json" because public access control lists (ACLs) are blocked by the BlockPublicAcls block public access setting. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>User: arn:aws:iam:::user/xenforo_user is not authorized to perform: s3: putObject on resource: "arn:aws:s3:::/internal_data/oembed_cache/1/1064-.json" because public access control lists (ACLs) are blocked by the BlockPublicAcls block public access setting.</Message><RequestId>**</RequestId><HostId></Error>

Is it a an expected behaviour to have public files from internal_data?
 
floriancw said:
Just starting using S3 buckets for data and internal_data with Xenforo.

I had to enable ACLs which is not recommended anymore, but more importantly, I found out that it is trying to enable public access for files in internal_data, which sounds counter intuitive at least to me:


Error executing "PutObject" on "https://****.s3.eu-west-3.amazonaws.com/internal_data/oembed_cache/1/1064-****.json"; AWS HTTP error: Client error: PUT https://****.s3.eu-west-3.amazonaws.com/internal_data/oembed_cache/1/1064-****c.json resulted in a 403 Forbidden response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>User: arn:aws:iam:::user/xen (truncated...) AccessDenied (client): User: arn:aws:iam:::user/xen** is not authorized to perform: s3: putObject on resource: "arn:aws:s3:::/internal_data/oembed_cache/1/1064-.json" because public access control lists (ACLs) are blocked by the BlockPublicAcls block public access setting. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>User: arn:aws:iam:::user/xenforo_user is not authorized to perform: s3: putObject on resource: "arn:aws:s3:::/internal_data/oembed_cache/1/1064-.json" because public access control lists (ACLs) are blocked by the BlockPublicAcls block public access setting.</Message><RequestId>**</RequestId><HostId></Error>

Is it a an expected behaviour to have public files from internal_data?
Click to expand...
I'm having the same issue. This tutorial works fine until you start putting the internal_data code in. You do need ACLs. You should go in and make your existing data publicly accessible and XenForo will use ACLs to automatically make future uploads public. I'm super irritated my reply was ignored and this tutorial didn't mention you need ACLs.

Even if you follow this tutorial correctly, it will refuse to work with the internal_data on S3. I have only gotten it to work if the internal_data is still local. The regular data folder works fine on S3.
 
Last edited:
I've followed the instructions but when I add the php code to the config file it breaks the forums and all I get is a white screen. Any thoughts on why this is happening? Thanks.

Code: 
#PHP:
#
#$s3 = function()
#{
#   return new \Aws\S3\S3Client([
#      'credentials' => [
#         'key' => 'XXXX',
#         'secret' => 'XXXX'
#      ],
#      'region' => 'tor1',
#      'version' => 'latest',
#      'endpoint' => 'https://tor1.digitaloceanspaces.com'
#   ]);
#};

#PHP:
#
#$config['fsAdapters']['data'] = function() use($s3)
#{
#   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data'>
#};

#PHP:
#
#$config['externalDataUrl'] = function($externalPath, $canonical)
#{
#   return 'https://xen-XXX.tor1.digitaloceanspaces.com/data/' . $externalPath;
#};
 
Last edited:
I still can't get anything but a white screen when adding the code the config. Just a few additional details: I'm running the latest version of Xenforo 2.3 with PHP 8.2.27. I do not have the add-on installed per the instructions. Any suggestions on how to get this working would be appreciated. Thanks.

Firefox shows a white screen, Chrome says http error 500.
 
Last edited:
In your config example you have a > at the end of one of the lines instead of ;. It’s probably that.
 
Specifically this line

Code: 
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data'>

Should be

Code: 
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data');
 
Chris D said:
Specifically this line

Code: 
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data'>

Should be

Code: 
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data');
Click to expand...
That was me not having Putty's window wide enough and it truncated my copying. The line in my config is actually as you say.

Code: 
#   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xen-XXX', 'data');
 
You must log in or register to reply here.

Similar threads

B
Using Digital Ocean Spaces or Amazon S3 for internal_data / attachments
Replies
3
Views
1K
Vlady
V
Back
Top Bottom