Bash script for backups

[Pauly]

Having recently migrated to a new server with more resources i am trying to update a bash script that takes a file dump of all domains to utilise multi thread compression to speed things up a bit
I have moved from centos to alma9 running centminmod so carrying scripts over from there

I use pbzip2 on my sql backups which are working fine and am trying to do the same on the file backups, can anyone help me modify or replace the below to do this please

#!/bin/bash
cd /home/nginx/domains/
for dir in *
do
base=$(basename "$dir")

if [ "$dir" == "backups" ] ; then
continue;
fi
backup_filename=$dir-`date +%F`
tar -czf /var/backups/"${backup_filename}.tar.gz" "$dir"
#echo $dir is now backed-up

done

I tried replacing the tar-czf line with a few things below and cant get it to work

pbzip2 -k /"${backup_filename}"�"$dir"
/usr/bin/pbzip2 -f /var/backups/"${backup_filename}.tar.gz" "$dir"
/usr/bin/pbzip2 -f /var/backups/*.$(/bin/date +%Y-%m-%d) "$dir"

I have to be honest and say i dont really understand bash so hoping one of you can point me in the right direction

 

[chillibear]

I think from a quick skim of the man page for pbzip2 that like most *NIX compression tools it is targeted at compressing single files - whilst it seems you can pass multiple files to pbzip2 it would simply compress each in turn - not compress them into one archive. I've no idea if your version of tar supports other compression options, generally it's gzip, bzip2 and xz, assuming you can't use pbzip via a -z -j -J etc switch you'll need to use it the old way... and pipe it through the compression tool and redirect that to a file, ie something like: tar -c *.jpg | pbzip2 > myjpegs.bz2

So I would expect you'd be wanting something more like.

#!/bin/bash
cd /home/nginx/domains/
for dir in *
do
  base=$(basename "$dir")
  if [ "$dir" == "backups" ] ; then
    continue;
  fi
  backup_filename=$dir-`date +%F`
  tar -c "$dir" | pbzip2 > /var/backups/"${backup_filename}.tar.bz2"
  #echo $dir is now backed-up
done

Caveat emptor and all that, I'm either a vanilla bourne shell or tcsh user myself. And you'll no doubt want to pass some options to pbzip2 to tweak it's CPU use etc.

 

[eva2000]

I have moved from centos to alma9 running centminmod so carrying scripts over from there

If you're running Centmin Mod, check out centmin.sh menu option 21, which can be for migration of data, but can also be a tool for backups https://centminmod.com/menu21-140.00beta01

centmin.sh menu option 21 is both a shell based menu with accompanying standalone command line script tool equivalent commands as well for each menu option so you can script your own backups as well.

--------------------------------------------------------
     Centmin Mod Data Management     
--------------------------------------------------------
1).   Manage SSH Keys
2).   Manage AWS CLI S3 Profile Credentials
3).   Migrate Centmin Mod Data To New Centmin Mod Server
4).   Backup Nginx Vhosts Data + MariaBackup MySQL Backups
5).   Backup Nginx Vhosts Data Only (no MariaDB MySQL backups)
6).   Backup MariaDB MySQL With MariaBackup Only (no Vhosts Data backups)
7).   Backup MariaDB MySQL With mysqldump only (no Vhosts Data backups)
8).   Transfer Directory Data To Remote Server Via SSH
9).   Transfer Directory Data To S3 Compatible Storage
10).  Transfer Files To S3 Compatible Storage
11).  Download S3 Compatible Stored Data To Server
12).  S3 To S3 Compatible Storage Transfers
13).  List S3 Storage Buckets
14).  Back to Main menu
--------------------------------------------------------
Enter option [ 1 - 14 ]
--------------------------------------------------------

 

[Pauly]

I think from a quick skim of the man page for pbzip2 that like most *NIX compression tools it is targeted at compressing single files - whilst it seems you can pass multiple files to pbzip2 it would simply compress each in turn - not compress them into one archive. I've no idea if your version of tar supports other compression options, generally it's gzip, bzip2 and xz, assuming you can't use pbzip via a -z -j -J etc switch you'll need to use it the old way... and pipe it through the compression tool and redirect that to a file, ie something like: tar -c *.jpg | pbzip2 > myjpegs.bz2

So I would expect you'd be wanting something more like.

#!/bin/bash
cd /home/nginx/domains/
for dir in *
do
  base=$(basename "$dir")
  if [ "$dir" == "backups" ] ; then
    continue;
  fi
  backup_filename=$dir-`date +%F`
  tar -c "$dir" | pbzip2 > /var/backups/"${backup_filename}.tar.bz2"
  #echo $dir is now backed-up
done

Caveat emptor and all that, I'm either a vanilla bourne shell or tcsh user myself. And you'll no doubt want to pass some options to pbzip2 to tweak it's CPU use etc.

Just wanted to revisit this and say thanks, works as it should with your suggestion and outputs a single compressed file for each domain, perfect !

 

[MySiteGuy]

Just wanted to revisit this and say thanks, works as it should with your suggestion and outputs a single compressed file for each domain, perfect !

If you're sticking with the method you have now, I recommend replacing pbzip2 with lbzip2 for much better speed.

 

[Pauly]

If you're sticking with the method you have now, I recommend replacing pbzip2 with lbzip2 for much better speed.

Thanks for the info, i installed lbzip2 today and swapped over to it
Did a quick check before and after and speed is greatly improved
DB backups dropped from 46 seconds to 33 and file backups dropped from 27 minutes to 20 minutes so quite a healthy saving !