Server backups / Duplicity

Backup

There is a regular backup ran from systemd; /etc/duplicity. It starts with a full dump of all MySQL tables. It is encrypted against a public key; the private key of which is held by the Trustees of the foundation.

Crontab kicks off a script;

  # monthly full, incrementals during the week.
  #
  3 3  1    * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh full
  3 3  2-31 * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh incremental

  # Half year retention for full; 1 months for the incrementals
  #
  1 1  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-but-n-full 6
  1 2  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-inc-of-but-n-full 1

The script: run.sh:

     #!/bin/sh
    set -e
    umask 077
    
    HOST="xxx.backup.host"
    
    DIR=/etc/duplicity
    W=incremental
    if [ $# != 0 ]; then
    	W=$1
    	shift
    fi
    T=
    if [ $W = full -o $W = incremental ];then
    	T=/
    	mysqldump --all-databases --single-transaction --quick --lock-tables=false  | gzip -9 > /var/lib/mysql-files/mysql-dump.gz
    fi
    
    # Verbose level 2 is errors and warnings; this way we skip
    # notices and quell all output if the backup is a success.
    #
    PASSPHRASE="YYYPASSWORD" \
         LANG=en_US.UTF8  LC_CTYPE=C HOME=$DIR GNUPGHOME=$DIR  \
         PYTHONWARNINGS="ignore::DeprecationWarning" \
    		python3 -W ignore::DeprecationWarning /usr/bin/duplicity $W $* \
    		\
    			-v 2 \
    			--hidden-encrypt-key XXXXX \
    			--hidden-encrypt-key YYYYY \
    			--sign-key           YYYYY \
    			--ssh-options="-i $DIR/backup.sftp -oUserKnownHostsFile=$DIR/knownhosts" \
    			--no-print-statistics \
    	\
    	--include /etc \
    	--include /usr/share/mediawiki \
    	--include /usr/share/wordpress \
    	--include /usr/local/makerspaceleiden-crm \
    	--exclude /var/lib/lxcfs \
    	--include /var/lib \
    	--include /var/www \
    	--include /var/log \
    	--exclude /dev \
    	--exclude /sys \
    	--exclude /run \
    	--exclude /tmp \
    	--exclude /snap \
    	--exclude /var/tmp \
    	--exclude /proc \
    	--exclude /swapfile \
    	--exclude /etc/duplicity/.cache \
    	\
    	$T \
    	sftp://msl@$HOST/backups 2>&1 |tee /var/log/last-duplcity-backup.new | grep -v DeprecationWarning | grep -v algorithm=hashes.SHA1
    	mv /var/log/last-duplcity-backup.new /var/log/duplicity.log
    	mv /var/log/duplicity.log.gz /var/log/duplicity.prevlog.gz || true
    	gzip /var/log/duplicity.log || true
    exit $?

Importing a new public key is done as follows

   cd /etc/duplicity
   cp XXXX/public-key-12345.gpg .
   gpg --homedir . --import public-key-12345.gpg .
   gpg --homedir . --edit-key XXXXXX
     trust 5
     save

And test by running it manually

  sudo /etc/duplicity/run.sh incremental

Safekeeping of private key

Example pgp key as QR

The code at https://github.com/dirkx/gpg-offline-batch-key- can be used to keep a backup off line.

It yields a printout like https://github.com/dirkx/gpg-offline-batch-key-/blob/master/sample-output.pdf.

Ransomware/targeted risk

This approach is not overly resistant against a targeted delete - as the sftp user can delete/modify files (as the retention is currently done from the 'source'). This is, to some extend, mitigated by zfs snapshots -- but not sufficiently at this time.

Restoring a file

To restore a single file - there is a script in

  /etc/duplicity/restore.sh

that is a wrapper for duplicity restore. Typical use is

   restore.sh restore --file-to-restore etc/foo/file.txt --time 2023-01-01

and it will put this in 'restored-file'.

History

2022/12 -- Changed to longer incremental runs; with only monthly full's.