Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

No permission to download

Chris D

XenForo developer
Staff member
Frankly, I have no idea. If the add-on is functioning normally and accessing attachments from internal_data works as it should there’s no reason why accessing anything else from internal_data wouldn’t work.

Clearly we don’t know the inner workings of the multi site add-on so we’d be working blind. It would be down to the add-on author to make it work with the abstracted file system rather than the other way around.
 

NandorHUN

Active member
Frankly, I have no idea. If the add-on is functioning normally and accessing attachments from internal_data works as it should there’s no reason why accessing anything else from internal_data wouldn’t work.

Clearly we don’t know the inner workings of the multi site add-on so we’d be working blind. It would be down to the add-on author to make it work with the abstracted file system rather than the other way around.
I deleted the internal_data part from the config.php and everything works.


The strange is that both domains write the files correctly in the folders. But the 2nd domain's (mydomain2.com/sitemap.php is "invalid file").
Manually it is fine.

And after I deleted the internal data from config it writes the same files but it works.
 

Chris D

XenForo developer
Staff member
I’d still have to refer you back to the add-on developer for assistance. When using the abstracted file system sitemaps work fine so whatever issue you’re seeing is likely caused by or not handled correctly by the multi site add-on.

Note that attachments are stored in internal_data so you’re very likely negating the benefits of offloading the file system to remote storage if you no longer have that configured.
 

NandorHUN

Active member
I’d still have to refer you back to the add-on developer for assistance. When using the abstracted file system sitemaps work fine so whatever issue you’re seeing is likely caused by or not handled correctly by the multi site add-on.

Note that attachments are stored in internal_data so you’re very likely negating the benefits of offloading the file system to remote storage if you no longer have that configured.
Attachments as pictures are stored on data folder no?

I tested it as added an image to a comment and it was uploaded to digital ocean, not the original server.

Or are you talking about other attachments like .rar files etc...?
 

Chris D

XenForo developer
Staff member
Thumbnails are stored in the data folder. Attachments (all types) are stored in the internal_data folder.
 

NandorHUN

Active member
Thumbnails are stored in the data folder. Attachments (all types) are stored in the internal_data folder.
Well, ok.
But guess, I cant do much if it doesnt work that way. Only other option is that each time I manually download the sitemap from Digital Ocean, upload it somewhere else in a separate folder on the site and link that back to Google Search Console.

On the long term maybe that will be the solution if I run out of space.

I wanted to set it up before I start the site, so that I dont have to migrate the data..... but just in case, if I need to do that I simply add the lines in the config.php and copy the files from internal_data to Digital Ocean's internal_data, no?
 

Chris D

XenForo developer
Staff member
Ideally the add-on developers would fix what is very likely a bug in their code.

I wanted to set it up before I start the site, so that I dont have to migrate the data..... but just in case, if I need to do that I simply add the lines in the config.php and copy the files from internal_data to Digital Ocean's internal_data, no?
That's right
 

HomerJ

Member
Am I correct in thinking that we only need to transfer the /data/ folder (and not the /internal_data/ folder too)?
 

Chris D

XenForo developer
Staff member
You can do one, the other or both. You just use only the required parts of the config in the guide.

Though, to be clear, if you only want one, you probably want internal_data. That is where the main attachment files are stored and therefore where the majority of your attachment data is stored. data only contains avatars and thumbnails.
 

HomerJ

Member
You can do one, the other or both. You just use only the required parts of the config in the guide.

Though, to be clear, if you only want one, you probably want internal_data. That is where the main attachment files are stored and therefore where the majority of your attachment data is stored. data only contains avatars and thumbnails.
Thanks. Just to be clear - it would just be the /internal_data/attachments/ dir right? Other dir's in internal_data seem to contain html files, php files etc.
 

Chris D

XenForo developer
Staff member
No, almost the entire internal_data directory would be hosted remotely. Though there is an exception to that which is stuff inside internal_data/code_cache would still be hosted locally.

So I'd transfer everything in internal_data over to the remote storage, then delete the code_cache directory on the remote side (as it won't be used).

Then you can delete everything inside internal_data except code_cache, the .htaccess and index.html. Although we'd obviously strongly recommend keeping a backup of the existing internal_data directory in case anything goes wrong.
 

HomerJ

Member
No, almost the entire internal_data directory would be hosted remotely. Though there is an exception to that which is stuff inside internal_data/code_cache would still be hosted locally.

So I'd transfer everything in internal_data over to the remote storage, then delete the code_cache directory on the remote side (as it won't be used).

Then you can delete everything inside internal_data except code_cache, the .htaccess and index.html. Although we'd obviously strongly recommend keeping a backup of the existing internal_data directory in case anything goes wrong.
So in actuality, if wanting to off-load everything, it's the entire data/ folder plus the entire internal_data/ folder (minus internal_data/code_cache if wanting to keep the bucket clean).

Got it, cheers :)
 

Chris D

XenForo developer
Staff member
Full House Ok GIF
 

briansol

Well-known member
I have internal_data/templates on both local and s3. neither has been updated in a long time (1 year plus) despite me upgrading xf a few times, and assuming addons would update templates as well.

Can i safely delete these? not sure where they are from.
sample:
S.0,L.0,xenforo.css.php
 

tonmo

Well-known member
Anything we need to do regarding this forthcoming change at AWS?

As noted in our prior notices sent on March 31, 2020 and August 20, 2020, we are updating all AWS Federal Information Processing Standard (FIPS) endpoints to a minimum of Transport Layer Security (TLS) 1.2 across all regions. Beginning March 31, 2021, if your client application cannot support TLS 1.2, it will result in connection failures to AWS FIPS endpoints [1]. To avoid an interruption in service, we encourage you to act now to ensure that you connect to AWS FIPS endpoints at TLS version 1.2. This change does not affect non-FIPS AWS endpoints.

AWS continues to directly notify impacted customers by their Personal Health Dashboard [2] and email. However, we are also sending this general notice (which is our third and final general notice), as customers connecting anonymously to AWS shared resources, such as through a public S3 bucket, would not have received direct notifications for those connections because we cannot identify anonymous connections.
 

Chris D

XenForo developer
Staff member
You need to make sure your server SSL library supports TLS 1.2. I believe this means as long as you're using OpenSSL 1.0.1 or above on your server (you can see this in phpinfo) it should have the required TLS 1.2 support.

For reference, I believe that's been standard since 2012. PayPal made this switch to requiring TLS 1.2 way back in 2017 and we explicitly warned about it in your Admin CP in XF 1.5 for a while if PayPal's endpoint detected that TLS 1.2 was unavailable.

On the whole, I suspect the sheer majority of customers will be absolutely fine and you'd potentially be experiencing issues already in other systems if TLS 1.2 wasn't available.
 

avclub

New member
Hello Gents,

I have managed to complete the S3 connection, files are getting uploaded to the bucket, however, I am experiencing an issue with the attachment thumbnails and the avatars. I have not yet migrated the old files, however, when uploading a new avatar, it comes up with a broken file. For the attachment thumbnails, there is the same issue, while when attaching a picture direcly to a post/thread, there is no issue at all. Also, when clicking on the image thumbnail, it opens in the lightbox without an issue.

Have tried changing image processing engines, and rebuilt the attachments thumbnails cache.
Any help would be more than welcome.
 
Top