Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

No permission to download

tonmo

Well-known member
Copy the Data. eg.

aws s3 cp internal-data s3://{yourbucketname}/internal-data/ --recursive

Was very easy.
I got the awscli2 working, thanks for the tips. Note, the example above refers to internal-data; the XF directory is internal_data.
Also - do you need to add any ACL permissions to the copy CLI command?
Now that I've moved all the files over (and purged cloudflare cache), I can see I still need some help with permissions. ...working it; almost there.
 

Hoffi

Well-known member
I got the awscli2 working, thanks for the tips. Note, the example above refers to internal-data; the XF directory is internal_data.
Oh, yes. My directory has a complete different name since it was possible to change it in the config. I assume since the beginning if I had this correct in mind, so I wrote it out of my head.
Now that I've moved all the files over (and purged cloudflare cache), I can see I still need some help with permissions. ...working it; almost there.
You can add a permission flag to the copy command, or split it up in two buckets. Both works.
 

tonmo

Well-known member
You can add a permission flag to the copy command, or split it up in two buckets. Both works.
Thanks. Rather than re-copying, AWS S3 has an option to "make public" any directories selected within a bucket; this seems to work fine:

1595764178032.png
 

Pawn Studios

Active member
@Chris D There's a typo in your guide specifically in the example code:


PHP:
$s3 = function()
{
    return new \Aws\S3\S3Client([
        'credentials' => [
            'key' => 'ABC',
            'secret' => '123'
        ],
        'region' => 'us-east-1',
        'version' => 'latest',
        'endpoint' => 'https://s3.eu-west-2.amazonaws.com'
    ]);
};

Note that the key and secret are what you noted down after setting up the IAM user earlier. The region can be inferred from the S3 endpoint URL.
In the example code, you have provided the region as 'region' => 'us-east-1'. It should rather be 'region' => ' eu-west-2 '.
 

hoks

Member
The add-on worked well with S3 for a few months, but something strange started happening 2 days ago

I get this fatal php error Uncaught Error: Class 'League\\Flysystem\\AwsS3v3\\AwsS3Adapter' not found in the config.php

The strange thing is it happens only on every other page load, not all the time.

Under what circumstances will the class go missing?

There were no changes to the server for weeks.
 
I have started to notice random 503 errors where no images load from digital ocean for a minute or two and then everything is fine again.

Still on xf 2.1 and have made no recent upgrades to the forum or addons.

Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://my-bucket.nyc3.digitalocean...2/62955-d44c0a03283de331ab7b2e11d32ac9c3.data"; AWS HTTP error: Server error: HEAD https://my-bucket.nyc3.digitaloceanspaces.com/internal_data/attachments/62/62955-d44c0a03283de331ab7b2e11d32ac9c3.data resulted in a 503 Service Unavailable response (server): 503 Service Unavailable - in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php at line 191
Code:
Aws\WrappedHttpHandler->parseError() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php at line 97
Aws\WrappedHttpHandler->Aws\{closure}() in src/vendor/guzzlehttp/promises/src/Promise.php at line 203
GuzzleHttp\Promise\Promise::callHandler() in src/vendor/guzzlehttp/promises/src/Promise.php at line 174
GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}() in src/vendor/guzzlehttp/promises/src/RejectedPromise.php at line 40
GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}() in src/vendor/guzzlehttp/promises/src/TaskQueue.php at line 47
GuzzleHttp\Promise\TaskQueue->run() in src/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php at line 98
GuzzleHttp\Handler\CurlMultiHandler->tick() in src/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php at line 125
GuzzleHttp\Handler\CurlMultiHandler->execute() in src/vendor/guzzlehttp/promises/src/Promise.php at line 246
GuzzleHttp\Promise\Promise->invokeWaitFn() in src/vendor/guzzlehttp/promises/src/Promise.php at line 223
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 267
GuzzleHttp\Promise\Promise->invokeWaitList() in src/vendor/guzzlehttp/promises/src/Promise.php at line 225
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 267
GuzzleHttp\Promise\Promise->invokeWaitList() in src/vendor/guzzlehttp/promises/src/Promise.php at line 225
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 62
GuzzleHttp\Promise\Promise->wait() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/AwsClientTrait.php at line 58
Aws\AwsClient->execute() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/S3ClientTrait.php at line 287
Aws\S3\S3Client->checkExistenceWithCommand() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/S3ClientTrait.php at line 272
Aws\S3\S3Client->doesObjectExist() in src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php at line 234
League\Flysystem\AwsS3v3\AwsS3Adapter->has() in src/vendor/league/flysystem/src/Filesystem.php at line 57
League\Flysystem\Filesystem->has()
call_user_func_array() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 431
League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 395
League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 128
League\Flysystem\EventableFilesystem\EventableFilesystem->has() in src/vendor/league/flysystem/src/MountManager.php at line 311
League\Flysystem\MountManager->has() in src/XF/Entity/AttachmentData.php at line 170
XF\Entity\AttachmentData->isDataAvailable() in src/XF/ControllerPlugin/Attachment.php at line 9
XF\ControllerPlugin\Attachment->displayAttachment() in src/XF/Pub/Controller/Attachment.php at line 43
XF\Pub\Controller\Attachment->actionIndex() in src/XF/Mvc/Dispatcher.php at line 350
XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 257
XF\Mvc\Dispatcher->dispatchFromMatch() in src/XF/Mvc/Dispatcher.php at line 113
XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 55
XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2190
XF\App->run() in src/XF.php at line 391
XF::runApp() in index.php at line 20
 

Jake B.

Well-known member
I have started to notice random 503 errors where no images load from digital ocean for a minute or two and then everything is fine again.

Still on xf 2.1 and have made no recent upgrades to the forum or addons.

That request is hitting DigitalOcean spaces directly, probably worth contacting their support about the issue
 
That request is hitting DigitalOcean spaces directly, probably worth contacting their support about the issue

Well, it's no longer sporadic as I have been seeing the last week or two. Everything has been down the last 15 minutes or so. Region NYC3 is now offline for maintenance according to Digital Ocean due to an unresolved incident.

Avatars are working because they are pulling from the CDN, and not only NYC3. I wonder if there is any adjustments I can make so attachments pull from somewhere else other than NYC3 for the timebeing.
 

Chromaniac

Well-known member
If the object storage itself is offline... it would not be possible to replicate it elsewhere (or locally on forum server) right? A CDN like Cloudflare would have prevented this one but it comes with other issues (bypasses guest restrictions... also the reason why xenforo does not use direct url for attachments that would have helped here otherwise).
 
If the object storage itself is offline... it would not be possible to replicate it elsewhere (or locally on forum server) right? A CDN like Cloudflare would have prevented this one but it comes with other issues (bypasses guest restrictions... also the reason why xenforo does not use direct url for attachments that would have helped here otherwise).

I have the CDN Digital Ocean offers turned on, so all my files are on other regions as well.

In any event, in the last minute or two NYC3 appears to be back online. So if the Avatars are loading from other regions, I thought it might be possible to try to pull the attachments in.

The status page still shows it as experiencing an incident.


In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted 4 hours ago. Aug 24, 2020 - 13:00 UTC
Scheduled
During the above window, our Storage Engineering team will be performing maintenance in our NYC3 region for our Spaces service, in order to improve performance and reliability.

Expected Impact:

Users may experience an increase in HTTP 503 errors, or periods of these errors, as well as increased latency in response times while working with objects in Spaces in NYC3, throughout the course of the maintenance. We will endeavour to keep these periods to a minimum.

If you have any questions or concerns, please reach out to us by opening up a support ticket on your account.
Posted 5 hours ago. Aug 24, 2020 - 12:00 UTC
 

frm

Well-known member
I am using it with Backblaze B2 S3 emulation api and it works great. Vultr should be fine as well!
Thanks. Currently using a dedicated server but would also like to essentially mirror a 3rd time. Vultr says S3 compatible, so just checking.

Do you know if you can keep the files on the same server have a 'backup' with object storage as opposed to just serving from object storage? I like redundancy.
 
Top