Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

[eDaddi]

AFAIK, there is no solution to caching images and attachments outside of Xenforo while protecting files.

I just got dinged nearly $500, unexpectedly, from Cloudflare for accelerated data transfer. From what I understand, if I put everything on Backblaze, I’d avoid this charge. So I’m back to looking at this. Is it possible to prevent images from a specific private node from being pushed to the CDN?

 

[JOGARA]

I just got dinged nearly $500, unexpectedly, from Cloudflare for accelerated data transfer. From what I understand, if I put everything on Bcakbkaze, I’d avoid this charge. So I’m back to looking at this. Is it possible to prevent images from a specific private node from being pushed to the CDN?

I don't don't have a clue what you are on about tbh.
Cloudflare should not be charging you for anything unless you subscribe to it.

How much data are you pushing through them?

 

[rdn]

I just got dinged nearly $500, unexpectedly, from Cloudflare for accelerated data transfer

Maybe this is from ARGO plan.

 

[eDaddi]

How much data are you pushing through them?

about 5GB apparently

Maybe this is from ARGO plan.
It must be, the invoice couldn’t be more vague.

 

[BubbaLovesCheese]

XenForo updated Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x with a new update entry:

Update for PHP 8.1 compatibility

Is Xenforo 2.2.7 compatible with PHP 8.1?

I thought it wasn't....

 

[Brogan]

Not yet fully compatible.

Fixed - PHP 8.1 Compatibility

Probably already known: It seems like XenForo ist currently not compatible with PHP 8.1 due to missing return type declarations on many methods, most notably on classes implementing core interfaes like \ArrayAccess This does generate error messages like An exception occurred: [ErrorException]...

xenforo.com

 

[Chris D]

We're aiming for initial support in XF 2.2.8.

Part of that effort flagged up this which I wanted to get released now so I didn't forget.

But, regardless, PHP 8.1 has not yet been released and it would be strongly ill-advised to even contemplate using PHP 8.1 in production until it has been through a few releases (and maybe even until we've been a through a few...)

 

[BubbaLovesCheese]

We're aiming for initial support in XF 2.2.8.

Part of that effort flagged up this which I wanted to get released now so I didn't forget.

But, regardless, PHP 8.1 has not yet been released and it would be strongly ill-advised to even contemplate using PHP 8.1 in production until it has been through a few releases (and maybe even until we've been a through a few...)

Gotcha, thanks.

That's what I sort of thought was happening, but I just wanted clarify and make sure I wasn't losing the plot.

Thanks.

 

[Rasmus Vind]

I like this guide, but I had a few problems.

• We need to URL redirect any instances of users linking to /data/ to my-cdn.com/data.
• We also need to rebuild master data if we have any uploaded styles that use ASSET: prefix. When rebuilt, the asset paths will be updated to the new data URL.

Aside from that it has been smooth sailing for me.

Good job.

 

[briansol]

How are you direct-linking to /data/ ? that should all be abstracted by attachment.php and so forth.

 

[Rasmus Vind]

Stuff like avatars are direct linked right? It’s not so uncommon to put someone’s avatar in a IMG.

 

[XenForo]

XenForo updated Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+ with a new update entry:

Updates Amazon AWS SDK to version 3.209.16

The last update to this add-on updated the Amazon AWS SDK to version 3.204.2 but it implied this may not be required due to it only adding support for PHP 8.1.

We are further updating the AWS library today to version 3.209.16.

While there are still no significant changes in the behaviour of this add-on itself we do recommend updating to ensure you're using a more recent version of the library.

Read the rest of this update entry...

 

[RenegadeMonster]

Just went to update this addon from 2.2.0 to 2.2.1 and found that after downloading and copying the files over the admin cp still shows 2.2.0 installed and no updates available.

 

[Chris D]

That can really only mean the files were uploaded to the wrong location or otherwise not uploaded. The addon.json file in the add-on I uploaded earlier contains the updated version string and version ID.

 

[RenegadeMonster]

Looks like something else is going on.

After I attempted the upgrade all images are broken.

When trying to do it again I get this error in FTP every time I try to copy the update over again or delete the plugin to copy it over freshly.



Tried rebooting the server, reconnecting to the FtP and trying again. Same error.

Very strange.

Edit:

It appears the issue was composer. After copping the updated addon over composer was running was deleting everything in the _vendor folder and then re adding it. It took a good 15 minutes to run through and brought down the whole forum with a 500 error for awhile.

But after it finished everything is working and I was able to see the update in the admin cp and upgrade the addon.

 

[Chris D]

That directory exists in the download.

Either way, I'm not feeling a great deal of confidence in your FTP client right now. I'd consider trying a different client entirely.

Are you not able to use the install/upgrade add-on from archive feature in the admin control panel?

Other than that, we have known overzealous server security software remove files or entire directories if, for some reason, it detects something objectionable in the filename or contents, so that could be blowing up.

 

[RenegadeMonster]

That directory exists in the download.

Either way, I'm not feeling a great deal of confidence in your FTP client right now. I'd consider trying a different client entirely.

Are you not able to use the install/upgrade add-on from archive feature in the admin control panel?

Other than that, we have known overzealous server security software remove files or entire directories if, for some reason, it detects something objectionable in the filename or contents, so that could be blowing up.

That makes me wonder if it was not composer. But I was able to see that every last file and folder inside of_vendor for the plugin was being removed one at a time. Until everything was gone. Once that happened I was able to copy the plugin back over and everything populated again and resolved itself.

Very weird, because I updated this addons the same way a few months back with no issues and I have updated all my other addons this way.

True, I could use archive feature, but I have always defaulted to just using FTP.

 

[azzurro]

In case anyone might run into similar issues, I saw these errors on page requests in the webserver log on upgrading from 2.2.0 to 2.2.1 (some server specific directory info removed):
stderr: PHP Fatal error: Uncaught Error: Call to undefined function Aws\\boolean_value() in web/src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Endpoint/UseFipsEndpoint/Configuration.php:14 stderr: Stack trace: stderr: #0 web/src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Endpoint/UseFipsEndpoint/ConfigurationProvider.php(174): Aws\\Endpoint\\UseFipsEndpoint\\Configuration->__construct(false) stderr: #1 web/src/vendor/guzzlehttp/promises/src/Promise.php(204): Aws\\Endpoint\\UseFipsEndpoint\\ConfigurationProvider::Aws\\Endpoint\\UseFipsEndpoint\\{closure}(Object(Aws\\Endpoint\\UseFipsEndpoint\\Exception\\ConfigurationException)) stderr: #2 web/src/vendor/guzzlehttp/promises/src/Promise.php(169): GuzzleHttp\\Promise\\Promise::callHandler(2, Object(Aws\\Endpoint\\UseFipsEndpoint\\Exception\\ConfigurationException), NULL) stderr: #3 /var/www/clients/client0/web1/web/src/vendor/guzzlehttp/promises/src/RejectedPromise.php(42): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}(Ob in web/src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Endpoint/UseFipsEndpoint/Configuration.php on line 14

Some image and video embeds weren't working on the pages. I tried a few things and running composer update in the XFAws directory I did see:
Updating dependencies
Package operations: 0 installs, 1 update, 0 removals
- Updating aws/aws-sdk-php (3.209.16 => 3.209.18): Loading from cache

And the errors go away and site works as expected. However, the XenForo health check currently complains that 17 files have 'unexpected contents', although a quick check they seem the same as the original:

src/addons/XFAws/composer.lock Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/athena/2017-05-18/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/ec2/2016-11-15/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/ec2/2016-11-15/paginators-1.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/endpoints.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/lakeformation/2017-03-31/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/metering.marketplace/2016-01-14/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/rbin/2021-06-15/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/robomaker/2018-06-29/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Ec2/Ec2Client.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/StreamWrapper.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Sdk.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_classmap.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_real.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_static.php Unexpected contents

src/addons/XFAws/_vendor/composer/ClassLoader.php Unexpected contents

src/addons/XFAws/_vendor/composer/installed.json Unexpected contents

 

[Jim Boy]

And the errors go away and site works as expected. However, the XenForo health check currently complains that 17 files have 'unexpected contents', although a quick check they seem the same as the original:

src/addons/XFAws/composer.lock Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/athena/2017-05-18/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/ec2/2016-11-15/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/ec2/2016-11-15/paginators-1.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/endpoints.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/lakeformation/2017-03-31/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/metering.marketplace/2016-01-14/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/rbin/2021-06-15/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/data/robomaker/2018-06-29/api-2.json.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Ec2/Ec2Client.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/S3/StreamWrapper.php Unexpected contents

src/addons/XFAws/_vendor/aws/aws-sdk-php/src/Sdk.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_classmap.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_real.php Unexpected contents

src/addons/XFAws/_vendor/composer/autoload_static.php Unexpected contents

src/addons/XFAws/_vendor/composer/ClassLoader.php Unexpected contents

src/addons/XFAws/_vendor/composer/installed.json Unexpected contents

Those files were updated in going from 3.209.16 => 3.209.18. You can see the change log at https://github.com/aws/aws-sdk-php/blob/master/CHANGELOG.md. Normally I would say that changes wouldn't affect XF, as XF wouldn't use more than 1% of the SDK, but these releases have "Improved error handling for failed writes and appends on unclosed streams" on S3, so if anything, will make it ever so slightly more robust.
As for your initial errors. Maybe a caching issue on your end? You use opcache? Its possible that a common library was cached that defined the boolean_value() function, however as the old sdk was more than 3 years old, that function may not have existed previously (fips didnt). Did you restart php (e.g. php-fpm) after the first update?

 

[Jim Boy]

Incidentally, this change now means you can use an ec2 role, if that is an option for you, rather than an IAM user, which is definitely a better practise.