Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

No permission to download
Overview Updates (4) Reviews (7) Discussion
Hoffi said:
Copy the Data. eg.

aws s3 cp internal-data s3://{yourbucketname}/internal-data/ --recursive

Was very easy.
Click to expand...
I got the awscli2 working, thanks for the tips. Note, the example above refers to internal-data; the XF directory is internal_data.
tonmo said:
Also - do you need to add any ACL permissions to the copy CLI command?
Click to expand...
Now that I've moved all the files over (and purged cloudflare cache), I can see I still need some help with permissions. ...working it; almost there.
 
tonmo said:
I got the awscli2 working, thanks for the tips. Note, the example above refers to internal-data; the XF directory is internal_data.
Click to expand...
Oh, yes. My directory has a complete different name since it was possible to change it in the config. I assume since the beginning if I had this correct in mind, so I wrote it out of my head.
tonmo said:
Now that I've moved all the files over (and purged cloudflare cache), I can see I still need some help with permissions. ...working it; almost there.
Click to expand...
You can add a permission flag to the copy command, or split it up in two buckets. Both works.
 
Hoffi said:
You can add a permission flag to the copy command, or split it up in two buckets. Both works.
Click to expand...
Thanks. Rather than re-copying, AWS S3 has an option to "make public" any directories selected within a bucket; this seems to work fine:

1595764178032.webp
 
@Chris D There's a typo in your guide specifically in the example code:


PHP: 
$s3 = function()
{
    return new \Aws\S3\S3Client([
        'credentials' => [
            'key' => 'ABC',
            'secret' => '123'
        ],
        'region' => 'us-east-1',
        'version' => 'latest',
        'endpoint' => 'https://s3.eu-west-2.amazonaws.com'
    ]);
};

Note that the key and secret are what you noted down after setting up the IAM user earlier. The region can be inferred from the S3 endpoint URL.
Click to expand...
In the example code, you have provided the region as 'region' => 'us-east-1'. It should rather be 'region' => ' eu-west-2 '.
 
The add-on worked well with S3 for a few months, but something strange started happening 2 days ago

I get this fatal php error Uncaught Error: Class 'League\\Flysystem\\AwsS3v3\\AwsS3Adapter' not found in the config.php

The strange thing is it happens only on every other page load, not all the time.

Under what circumstances will the class go missing?

There were no changes to the server for weeks.
 
I have started to notice random 503 errors where no images load from digital ocean for a minute or two and then everything is fine again.

Still on xf 2.1 and have made no recent upgrades to the forum or addons.

Aws\S3\Exception\S3Exception: Error executing "HeadObject" on "https://my-bucket.nyc3.digitalocean...2/62955-d44c0a03283de331ab7b2e11d32ac9c3.data"; AWS HTTP error: Server error: HEAD https://my-bucket.nyc3.digitaloceanspaces.com/internal_data/attachments/62/62955-d44c0a03283de331ab7b2e11d32ac9c3.data resulted in a 503 Service Unavailable response (server): 503 Service Unavailable - in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php at line 191
Code: 
Aws\WrappedHttpHandler->parseError() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php at line 97
Aws\WrappedHttpHandler->Aws\{closure}() in src/vendor/guzzlehttp/promises/src/Promise.php at line 203
GuzzleHttp\Promise\Promise::callHandler() in src/vendor/guzzlehttp/promises/src/Promise.php at line 174
GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}() in src/vendor/guzzlehttp/promises/src/RejectedPromise.php at line 40
GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}() in src/vendor/guzzlehttp/promises/src/TaskQueue.php at line 47
GuzzleHttp\Promise\TaskQueue->run() in src/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php at line 98
GuzzleHttp\Handler\CurlMultiHandler->tick() in src/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php at line 125
GuzzleHttp\Handler\CurlMultiHandler->execute() in src/vendor/guzzlehttp/promises/src/Promise.php at line 246
GuzzleHttp\Promise\Promise->invokeWaitFn() in src/vendor/guzzlehttp/promises/src/Promise.php at line 223
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 267
GuzzleHttp\Promise\Promise->invokeWaitList() in src/vendor/guzzlehttp/promises/src/Promise.php at line 225
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 267
GuzzleHttp\Promise\Promise->invokeWaitList() in src/vendor/guzzlehttp/promises/src/Promise.php at line 225
GuzzleHttp\Promise\Promise->waitIfPending() in src/vendor/guzzlehttp/promises/src/Promise.php at line 62
GuzzleHttp\Promise\Promise->wait() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/AwsClientTrait.php at line 58
Aws\AwsClient->execute() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/S3ClientTrait.php at line 287
Aws\S3\S3Client->checkExistenceWithCommand() in src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/S3ClientTrait.php at line 272
Aws\S3\S3Client->doesObjectExist() in src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php at line 234
League\Flysystem\AwsS3v3\AwsS3Adapter->has() in src/vendor/league/flysystem/src/Filesystem.php at line 57
League\Flysystem\Filesystem->has()
call_user_func_array() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 431
League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 395
League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall() in src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php at line 128
League\Flysystem\EventableFilesystem\EventableFilesystem->has() in src/vendor/league/flysystem/src/MountManager.php at line 311
League\Flysystem\MountManager->has() in src/XF/Entity/AttachmentData.php at line 170
XF\Entity\AttachmentData->isDataAvailable() in src/XF/ControllerPlugin/Attachment.php at line 9
XF\ControllerPlugin\Attachment->displayAttachment() in src/XF/Pub/Controller/Attachment.php at line 43
XF\Pub\Controller\Attachment->actionIndex() in src/XF/Mvc/Dispatcher.php at line 350
XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 257
XF\Mvc\Dispatcher->dispatchFromMatch() in src/XF/Mvc/Dispatcher.php at line 113
XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 55
XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2190
XF\App->run() in src/XF.php at line 391
XF::runApp() in index.php at line 20
Click to expand...
 
RenegadeMonster said:
I have started to notice random 503 errors where no images load from digital ocean for a minute or two and then everything is fine again.

Still on xf 2.1 and have made no recent upgrades to the forum or addons.
Click to expand...

That request is hitting DigitalOcean spaces directly, probably worth contacting their support about the issue
 
Jake B. said:
That request is hitting DigitalOcean spaces directly, probably worth contacting their support about the issue
Click to expand...

Well, it's no longer sporadic as I have been seeing the last week or two. Everything has been down the last 15 minutes or so. Region NYC3 is now offline for maintenance according to Digital Ocean due to an unresolved incident.

Avatars are working because they are pulling from the CDN, and not only NYC3. I wonder if there is any adjustments I can make so attachments pull from somewhere else other than NYC3 for the timebeing.
 
If the object storage itself is offline... it would not be possible to replicate it elsewhere (or locally on forum server) right? A CDN like Cloudflare would have prevented this one but it comes with other issues (bypasses guest restrictions... also the reason why xenforo does not use direct url for attachments that would have helped here otherwise).
 
Chromaniac said:
If the object storage itself is offline... it would not be possible to replicate it elsewhere (or locally on forum server) right? A CDN like Cloudflare would have prevented this one but it comes with other issues (bypasses guest restrictions... also the reason why xenforo does not use direct url for attachments that would have helped here otherwise).
Click to expand...

I have the CDN Digital Ocean offers turned on, so all my files are on other regions as well.

In any event, in the last minute or two NYC3 appears to be back online. So if the Avatars are loading from other regions, I thought it might be possible to try to pull the attachments in.

The status page still shows it as experiencing an incident.

DigitalOcean Status

Welcome to DigitalOcean's home for real-time and historical data on system performance.
status.digitalocean.com status.digitalocean.com

In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted 4 hours ago. Aug 24, 2020 - 13:00 UTC
Scheduled
During the above window, our Storage Engineering team will be performing maintenance in our NYC3 region for our Spaces service, in order to improve performance and reliability.

Expected Impact:

Users may experience an increase in HTTP 503 errors, or periods of these errors, as well as increased latency in response times while working with objects in Spaces in NYC3, throughout the course of the maintenance. We will endeavour to keep these periods to a minimum.

If you have any questions or concerns, please reach out to us by opening up a support ticket on your account.
Posted 5 hours ago. Aug 24, 2020 - 12:00 UTC
Click to expand...
 
Chromaniac said:
I am using it with Backblaze B2 S3 emulation api and it works great. Vultr should be fine as well!
Click to expand...
Thanks. Currently using a dedicated server but would also like to essentially mirror a 3rd time. Vultr says S3 compatible, so just checking.

Do you know if you can keep the files on the same server have a 'backup' with object storage as opposed to just serving from object storage? I like redundancy.
 
You must log in or register to reply here.
Top Bottom