backup of important directories in /data and /internal_data

[rebelde]

Just wanting to make sure that I don't exclude anything important...

In the /data and internal_data, directories, I think I only need to back up:

/data/avatars
/data/profile_banners
/internal_data/attachments

Am I missing anything? The rest seems to be files, caches and thumbnails.

Thanks!

----------------------


For the record, here are those two directories:

[root@forum xenforo]# ls data/ -1
attachments
avatars
index.html
profile_banners

[root@forum xenforo]# ls internal_data/ -1
addon_batch
attachments
code_cache
file_check
image_cache
index.html
install-lock.php
keys
oembed_cache
page_cache
sitemaps
temp
templates
xen.log

And, in case it helps anyone, the relevant lines for aws s3 sync in crontab:

28 11 * * * /root/.local/bin/aws s3 sync /xenforo2/data/avatars s3://my-forum-backup/xenforo2/data/avatars/
28 11 * * * /root/.local/bin/aws s3 sync /xenforo2/data/profile_banners s3://my-forum-backup/xenforo2/data/profile_banners/
33 11 * * * /root/.local/bin/aws s3 sync /xenforo2/internal_data/attachments s3://my-forum-backup/xenforo2/internal_data/attachments/

 

[digitalpoint]

You are best off backing up the entire data directory (anything there, you want to keep). If you don't use certain things (like allowing users to upload video), those directories will be empty. But ultimately, anything that is there, you want to backup.

For internal_data, attachments and keys are going to be the subdirectories that would cause a problem if they were lost.

 

[TPerry]

Errr... if you are using S3... I'm pretty sure your data isn't being kept locally... but on your S3 instance... so THAT is where you need to be backing up from, not your local directories?
That was the whole purpose of using S3/R2...to off-load that content to a remote site if I understood the purpose correctly.
I am assuming you are inquiring what subs under the /data and /internal_data on your S3/R2 instance needs to be backed up?

 

[digitalpoint]

Errr... if you are using S3... I'm pretty sure your data isn't being kept locally... but on your S3 instance... so THAT is where you need to be backing up from, not your local directories..
That was the whole purpose of using S3/R2...to off-load that content to a remote site.

From the looks of it, he's backing up to S3, not from S3. But ya... not how I would do it, but the question wasn't, "What's the best way to store data/internal_data)?", so I didn't go down the road of answering a question someone didn't ask.

 

[TPerry]

yeah.. for the "space" required.. I just do a full backup of /data and /internal_data. It generally isn't going to make that big of a difference.

 

[rebelde]

Thanks for the responses!

Right, backing up TO s3 from my server.

For internal_data, attachments and keys are going to be the subdirectories that would cause a problem if they were lost.

OK, so add keys. Thanks.

the question wasn't, "What's the best way to store data/internal_data)?", so I didn't go down the road of answering a question someone didn't ask

I can accept the critique, I think. How would you do it? My goal is to (incrementally) back up the files off the server somewhere in case of emergency. This method seems simple and effective.

 

[TPerry]

Personally.. and yes, it requires more "space"... but I export my site DB into my root directory.. then I simply back up the entire root directory to an archive... then I delete that exported DB in my root, and then have automation used to sync the directory that I exported that backup to to my local computer drive, an attached external storage drive and then with a remote NAS.

 

[digitalpoint]

I can accept the critique, I think. How would you do it? My goal is to (incrementally) back up the files off the server somewhere in case of emergency. This method seems simple and effective.

XF2 has an abstracted filesystem that allows you to keep your stuff in things like S3 (not just backup to it). Personally, I use Cloudflare's R2 system.

Announcing Cloudflare R2 Storage: Rapid and Reliable Object Storage, minus the egress fees

Introducing Cloudflare’s S3-compatible Object Storage service, with zero egress bandwidth charges and automatic migration from S3-compatible services.

blog.cloudflare.com

The core of what makes Object Storage great is reliability — we designed R2 for data durability and resilience at its core. R2 will provide 99.999999999% (eleven 9’s) of annual durability, which describes the likelihood of data loss. If you store 1,000,000 objects on R2, you can expect to lose one once every 100,000 years — the same level of durability as other major providers. R2 will be resistant to regional failures, replicating objects multiple times for high availability.

Pricing is more than reasonable (free for the first 10GB, $0.015 per GB per month after that). So for example if you had 50GB worth of attachments/avatars, etc. offloaded from your server to the cloud (R2), your cost would be $0.60 per month and then you don't need to worry about backups.

 

[rebelde]

Personally, I use Cloudflare's R2 system.

Interesting. I like the idea: CDN + backup that isn't too expensive.

But one of the aspects of AWS S3 that I like is "versioning", where they keep old copies of the files. If someone were to delete the /internal_data directory on my server, S3 would not delete its copies. Does R2 offer a similar backup? I can't find that it does. All I see is Deleting objects from a bucket is irreversible., which is not what I want.

 

[digitalpoint]

Ya, R2 doesn’t protect against someone deleting something accidentally. I did read that versioning is coming, but it’s not available yet.

 

[fly]

Thanks for the responses!

Right, backing up TO s3 from my server.


OK, so add keys. Thanks.


I can accept the critique, I think. How would you do it? My goal is to (incrementally) back up the files off the server somewhere in case of emergency. This method seems simple and effective.

If you're running the server on AWS EC2 (or Lightsail), just take regular snapshots. If you want diffs, gzip it locally and send it to S3 with a timestamp.

 

[⭐ Alex ⭐]

You are best off backing up the entire data directory (anything there, you want to keep). If you don't use certain things (like allowing users to upload video), those directories will be empty. But ultimately, anything that is there, you want to backup.

For internal_data, attachments and keys are going to be the subdirectories that would cause a problem if they were lost.

Is the keys folder something that stores generated XF API keys or something? I would assume those go into the database. I don't have a keys folder so I'm wondering what goes inside there.

EDIT: Nvm, Email DKIM key. Thanks.

Make data & internal_data filesystem adaptor more granular

The problem: You want to store attachments and avatars in a non-local filesystem. The solution: There are currently 3 locations you can override filesystem adaptors: data, internal-data and code-cache. So to solve "the problem" you can override data and internal-data to use something like...

xenforo.com