Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.x

No permission to download

Chris D

XenForo developer
Staff member
Frankly, I have no idea. If the add-on is functioning normally and accessing attachments from internal_data works as it should there’s no reason why accessing anything else from internal_data wouldn’t work.

Clearly we don’t know the inner workings of the multi site add-on so we’d be working blind. It would be down to the add-on author to make it work with the abstracted file system rather than the other way around.
 

NandorHUN

Active member
Frankly, I have no idea. If the add-on is functioning normally and accessing attachments from internal_data works as it should there’s no reason why accessing anything else from internal_data wouldn’t work.

Clearly we don’t know the inner workings of the multi site add-on so we’d be working blind. It would be down to the add-on author to make it work with the abstracted file system rather than the other way around.
I deleted the internal_data part from the config.php and everything works.


The strange is that both domains write the files correctly in the folders. But the 2nd domain's (mydomain2.com/sitemap.php is "invalid file").
Manually it is fine.

And after I deleted the internal data from config it writes the same files but it works.
 

Chris D

XenForo developer
Staff member
I’d still have to refer you back to the add-on developer for assistance. When using the abstracted file system sitemaps work fine so whatever issue you’re seeing is likely caused by or not handled correctly by the multi site add-on.

Note that attachments are stored in internal_data so you’re very likely negating the benefits of offloading the file system to remote storage if you no longer have that configured.
 

NandorHUN

Active member
I’d still have to refer you back to the add-on developer for assistance. When using the abstracted file system sitemaps work fine so whatever issue you’re seeing is likely caused by or not handled correctly by the multi site add-on.

Note that attachments are stored in internal_data so you’re very likely negating the benefits of offloading the file system to remote storage if you no longer have that configured.
Attachments as pictures are stored on data folder no?

I tested it as added an image to a comment and it was uploaded to digital ocean, not the original server.

Or are you talking about other attachments like .rar files etc...?
 

Chris D

XenForo developer
Staff member
Thumbnails are stored in the data folder. Attachments (all types) are stored in the internal_data folder.
 

NandorHUN

Active member
Thumbnails are stored in the data folder. Attachments (all types) are stored in the internal_data folder.
Well, ok.
But guess, I cant do much if it doesnt work that way. Only other option is that each time I manually download the sitemap from Digital Ocean, upload it somewhere else in a separate folder on the site and link that back to Google Search Console.

On the long term maybe that will be the solution if I run out of space.

I wanted to set it up before I start the site, so that I dont have to migrate the data..... but just in case, if I need to do that I simply add the lines in the config.php and copy the files from internal_data to Digital Ocean's internal_data, no?
 

Chris D

XenForo developer
Staff member
Ideally the add-on developers would fix what is very likely a bug in their code.

I wanted to set it up before I start the site, so that I dont have to migrate the data..... but just in case, if I need to do that I simply add the lines in the config.php and copy the files from internal_data to Digital Ocean's internal_data, no?
That's right
 

Chris D

XenForo developer
Staff member
You can do one, the other or both. You just use only the required parts of the config in the guide.

Though, to be clear, if you only want one, you probably want internal_data. That is where the main attachment files are stored and therefore where the majority of your attachment data is stored. data only contains avatars and thumbnails.
 

HomerJ

Member
You can do one, the other or both. You just use only the required parts of the config in the guide.

Though, to be clear, if you only want one, you probably want internal_data. That is where the main attachment files are stored and therefore where the majority of your attachment data is stored. data only contains avatars and thumbnails.
Thanks. Just to be clear - it would just be the /internal_data/attachments/ dir right? Other dir's in internal_data seem to contain html files, php files etc.
 

Chris D

XenForo developer
Staff member
No, almost the entire internal_data directory would be hosted remotely. Though there is an exception to that which is stuff inside internal_data/code_cache would still be hosted locally.

So I'd transfer everything in internal_data over to the remote storage, then delete the code_cache directory on the remote side (as it won't be used).

Then you can delete everything inside internal_data except code_cache, the .htaccess and index.html. Although we'd obviously strongly recommend keeping a backup of the existing internal_data directory in case anything goes wrong.
 

HomerJ

Member
No, almost the entire internal_data directory would be hosted remotely. Though there is an exception to that which is stuff inside internal_data/code_cache would still be hosted locally.

So I'd transfer everything in internal_data over to the remote storage, then delete the code_cache directory on the remote side (as it won't be used).

Then you can delete everything inside internal_data except code_cache, the .htaccess and index.html. Although we'd obviously strongly recommend keeping a backup of the existing internal_data directory in case anything goes wrong.
So in actuality, if wanting to off-load everything, it's the entire data/ folder plus the entire internal_data/ folder (minus internal_data/code_cache if wanting to keep the bucket clean).

Got it, cheers :)
 

Chris D

XenForo developer
Staff member
Full House Ok GIF
 

briansol

Well-known member
I have internal_data/templates on both local and s3. neither has been updated in a long time (1 year plus) despite me upgrading xf a few times, and assuming addons would update templates as well.

Can i safely delete these? not sure where they are from.
sample:
S.0,L.0,xenforo.css.php
 
Top