Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Overview Updates (4) Reviews (7) Discussion
xenforo.com

Image Optimizer for XF 2.0

Add-on have provide a powerful settings in ACP options. You MUST settings option before running optimize images. All files will be overwrite; you cannot revert the file. DESCRIPTION Optimizing all images which stored in XenForo to lower size...
xenforo.com xenforo.com
 
above addon does have a lot of active users and seems to have good reviews. delayed conversion is actually good in my opinion. gives you better control over timing it during less active hours when server has more available resources.
 
Chromaniac said:
above addon does have a lot of active users and seems to have good reviews. delayed conversion is actually good in my opinion. gives you better control over timing it during less active hours when server has more available resources.
Click to expand...
How do you backup your R2 object storage?
 
right. i haven't considered doing it yet but one rclone command would basically copy the entire data to another cloud storage or even locally or on onedrive.
 
Ever since I enabled R2, I end up getting these errors occasionally. I wonder if R2 basically times out sometimes.

Code: 
InvalidArgumentException: Must pass valid resource in src/XF/Http/ResponseStream.php:16
Generated by: Unknown account Oct 26, 2022 at 1:41 AM
Stack trace
#0 src/XF/Http/Response.php(247): XF\Http\ResponseStream->__construct(false, 178288)
#1 src/XF/Pub/View/Attachment/View.php(26): XF\Http\Response->responseStream(false, 178288)
#2 src/XF/Mvc/Renderer/AbstractRenderer.php(91): XF\Pub\View\Attachment\View->renderRaw()
#3 src/XF/Mvc/Renderer/Raw.php(41): XF\Mvc\Renderer\AbstractRenderer->renderViewObject('XF:Attachment\\V...', '', Array)
#4 src/XF/Mvc/Dispatcher.php(458): XF\Mvc\Renderer\Raw->renderView('XF:Attachment\\V...', '', Array)
#5 src/XF/Mvc/Dispatcher.php(440): XF\Mvc\Dispatcher->renderView(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#6 src/XF/Mvc/Dispatcher.php(400): XF\Mvc\Dispatcher->renderReply(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#7 src/XF/Mvc/Dispatcher.php(58): XF\Mvc\Dispatcher->render(Object(XF\Mvc\Reply\View), 'raw')
#8 src/XF/App.php(2344): XF\Mvc\Dispatcher->run()
#9 src/XF.php(512): XF\App->run()
#10 index.php(20): XF::runApp('XF\\Pub\\App')
#11 {main}

Googling this indicates this might be connected to Redis.
 
briansol said:
yes, you will need to manually move the old stuff up to s3. There's a tool called s3cmd that will let you send it up to the bucket from your server command line.
If you've done, that, make sure permissions are good. try to load the url in the browser alone and if you see an XML doc, your permissions are off.

Examples of s3cmd 2.x Usage with DigitalOcean Spaces :: DigitalOcean Product Documentation
Click to expand...
I am trying to install this. I now have s2cmd installed, however I can't figure how (or what) to copy up to DigitalOcean.

If I go to that link I get a 404 error.

Any tips for moving my stuff up to them? (and what I need to send up to them) I am on Xenforo 2.2.11

Thanks!
 
I had this working fine on a previous version using digital ocean method. I am doing a fresh install using the latest version of Xenforo. As soon as i added code to config I am unable to log into administrative panel. Any help would be appreciated. Here's my config file...


<?php

$config['db']['host'] = 'localhost';
$config['db']['port'] = '3306';
$config['db']['username'] = 'xxxxxxxxxxxxxx';
$config['db']['password'] = 'xxxxxxxxxxxxxx';
$config['db']['dbname'] = 'xxxxxxxxxxxxxx';
$config['fullUnicode'] = true;


$s3 = function()
{
return new \Aws\S3\S3Client([
'credentials' => [
'key' => 'xxxxxxxxxxxxxx',
'secret' => 'xxxxxxxxxxxxxx'
],
'region' => 'nyc3',
'version' => 'latest',
'endpoint' => 'https://nyc3.digitaloceanspaces.com/'
]);
};


$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'data');
};


$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.nyc3.digitaloceanspaces.com/data/' . $externalPath;
};


$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'internal_data');
};
 
Chris D said:
The typical Cloud plans don't utilise remote object storage. Files are locally stored (on high performance SSDs).
Click to expand...
Is this something someone utilizing a XF Cloud plan could implement? (High-resolution photo and video uploads would consume the storage limits on Cloud plans rather quickly)
 
We’re able to provision additional storage for Cloud plans utilising off-server storage, although this is typically for Business level plans that otherwise fit comfortably in the usage limits for traffic.

This doesn’t use object storage but essentially network attached disks in the data centre.

There’s an additional monthly fee involved and we can discuss requirements if you contact us.
 
It's been over a year since I asked and maybe I missed it but is there still no solution for using S3 (or any one of the cloud services) to keep private forums attachments private?

Is there a reason why a simple IF/ELSE statement couldn't change the path from a cloud src to local host src for private forums pics/attachments?
 
I have tried in vain for hours to get this working with Backblaze B2 but I am drawing a complete blank. Followed the instructions to the letter, config looks correct and thumbnails and avatars load just fine - it seems the only problem is with the full attachments.

I did notice @Chromaniac had the same error sporadically, and I also have Redis enabled (via the Xon addon) but disabling it doesn't fix it.

I've verified the files are uploaded (working URL example HERE) and the CloudFlare redirect is in place, but just cannot get past this error:

Code: 
InvalidArgumentException: Must pass valid resource in src/XF/Http/ResponseStream.php:18

#0 src/XF/Http/Response.php(250): XF\Http\ResponseStream->__construct(false, 655793)
#1 src/XF/Pub/View/Attachment/View.php(30): XF\Http\Response->responseStream(false, 655793)
#2 src/XF/Mvc/Renderer/AbstractRenderer.php(91): XF\Pub\View\Attachment\View->renderRaw()
#3 src/XF/Mvc/Renderer/Raw.php(39): XF\Mvc\Renderer\AbstractRenderer->renderViewObject('XF:Attachment\\V...', '', Array)
#4 src/XF/Mvc/Dispatcher.php(460): XF\Mvc\Renderer\Raw->renderView('XF:Attachment\\V...', '', Array)
#5 src/XF/Mvc/Dispatcher.php(442): XF\Mvc\Dispatcher->renderView(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#6 src/XF/Mvc/Dispatcher.php(402): XF\Mvc\Dispatcher->renderReply(Object(XF\Mvc\Renderer\Raw), Object(XF\Mvc\Reply\View))
#7 src/XF/Mvc/Dispatcher.php(60): XF\Mvc\Dispatcher->render(Object(XF\Mvc\Reply\View), 'raw')
#8 src/XF/App.php(2483): XF\Mvc\Dispatcher->run()
#9 src/XF.php(524): XF\App->run()
#10 index.php(20): XF::runApp('XF\\Pub\\App')
#11 {main}

Redacted relevant config.php:
PHP: 
$s3 = function()
{
   return new \Aws\S3\S3Client([
      'credentials' => [
         'key' => '[redacted]',
         'secret' => '[redacted]'
      ],
      'region' => 'eu-central-003',
      'version' => 'latest',
      'endpoint' => 'https://s3.eu-central-003.backblazeb2.com'
   ]);
};

$config['fsAdapters']['data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'msc-attachments', 'data');
};

$config['externalDataUrl'] = function($externalPath, $canonical)
{
   return 'https://cdn.micra.org.uk/data/' . $externalPath;
};

$config['fsAdapters']['internal-data'] = function() use($s3)
{
   return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'msc-attachments', 'internal_data');
};

Any help would be greatly appreciated as this is driving me mad!

Edited to add: I can also upload new files and they get correctly placed into the bucket, XF just can't show them!
 
This is probably due for a PHP 8.2 update. As far as I can tell, a simple composer update gets this working on 8.2.

Edit: For anyone else reading this: as of writing (2022-12-13), you still shouldn't be using 8.2 in production unless you're willing to debug and fix major issues on your own. XF 2.2.12 won't run on PHP 8.2.
 
Last edited:
Hey @Chris D - attachments are working and going into digitalocean. However, the avatars are not. What am I doing wrong?

$s3 = function()
{
return new \Aws\S3\S3Client([
'credentials' => [
'key' => 'XXXXXX',
'secret' => 'XXXXXXXXXXXXXXX'
],
'region' => 'nyc3',
'version' => 'latest',
'endpoint' => 'https://nyc3.digitaloceanspaces.com'
]);
};
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xfuploads', 'data');
};
$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xfuploads', 'internal_data');
};
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xfuploads.nyc3.digitaloceanspaces.com/data' . $externalPath;
 
This:

PHP: 
return 'https://xfuploads.nyc3.digitaloceanspaces.com/data' . $externalPath;

Should probably be:

PHP: 
return 'https://xfuploads.nyc3.digitaloceanspaces.com/data/' . $externalPath;

(Missing slash after data)
 
Chris D said:
This:

PHP: 
return 'https://xfuploads.nyc3.digitaloceanspaces.com/data' . $externalPath;

Should probably be:

PHP: 
return 'https://xfuploads.nyc3.digitaloceanspaces.com/data/' . $externalPath;

(Missing slash after data)
Click to expand...
oh man - Ive been starring at this for about 5 hours. This fixed it. Thanks!
 
You must log in or register to reply here.

Similar threads

B
Using Digital Ocean Spaces or Amazon S3 for internal_data / attachments
Replies
3
Views
1K
Vlady
V
Top Bottom