Server backups / Duplicity

Backup

There is a regular backup ran from systemd; /etc/duplicity. It starts with a full dump of all MySQL tables. It is encrypted against a public key; the private key of which is held by the Trustees of the foundation.

Crontab kicks off a script;

  # monthly full, incrementals during the week.
  #
  3 3  1    * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh full
  3 3  2-31 * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh incremental

  # Half year retention for full; 1 months for the incrementals
  #
  1 1  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-but-n-full 6
  1 2  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-inc-of-but-n-full 1

The script: run.sh:

     #!/bin/sh
    set -e
    umask 077
    
    HOST="xxx.backup.host"
    
    DIR=/etc/duplicity
    W=incremental
    if [ $# != 0 ]; then
    	W=$1
    	shift
    fi
    T=
    if [ $W = full -o $W = incremental ];then
    	T=/
    	mysqldump --all-databases --single-transaction --quick --lock-tables=false  | gzip -9 > /var/lib/mysql-files/mysql-dump.gz
    fi
    
    # Verbose level 2 is errors and warnings; this way we skip
    # notices and quell all output if the backup is a success.
    #
    PASSPHRASE="YYYPASSWORD" \
         LANG=en_US.UTF8  LC_CTYPE=C HOME=$DIR GNUPGHOME=$DIR  \
         PYTHONWARNINGS="ignore::DeprecationWarning" \
    		python3 -W ignore::DeprecationWarning /usr/bin/duplicity $W $* \
    		\
    			-v 2 \
    			--hidden-encrypt-key XXXXX \
    			--hidden-encrypt-key YYYYY \
    			--sign-key           YYYYY \
    			--ssh-options="-i $DIR/backup.sftp -oUserKnownHostsFile=$DIR/knownhosts" \
    			--no-print-statistics \
    	\
    	--include /etc \
    	--include /usr/share/mediawiki \
    	--include /usr/share/wordpress \
    	--include /usr/local/makerspaceleiden-crm \
    	--exclude /var/lib/lxcfs \
    	--include /var/lib \
    	--include /var/www \
    	--include /var/log \
    	--exclude /dev \
    	--exclude /sys \
    	--exclude /run \
    	--exclude /tmp \
    	--exclude /snap \
    	--exclude /var/tmp \
    	--exclude /proc \
    	--exclude /swapfile \
    	--exclude /etc/duplicity/.cache \
    	\
    	$T \
    	sftp://msl@$HOST/backups 2>&1 |tee /var/log/last-duplcity-backup.new | grep -v DeprecationWarning | grep -v algorithm=hashes.SHA1
    	mv /var/log/last-duplcity-backup.new /var/log/duplicity.log
    	mv /var/log/duplicity.log.gz /var/log/duplicity.prevlog.gz || true
    	gzip /var/log/duplicity.log || true
    exit $?

Importing a new public key is done as follows

   cd /etc/duplicity
   cp XXXX/public-key-12345.gpg .
   gpg --homedir . --import public-key-12345.gpg .
   gpg --homedir . --edit-key XXXXXX
     trust 5
     save

And test by running it manually

  sudo /etc/duplicity/run.sh incremental

Safekeeping of private key

Example pgp key as QR

The code at https://github.com/dirkx/gpg-offline-batch-key- can be used to keep a backup off line.

It yields a printout like https://github.com/dirkx/gpg-offline-batch-key-/blob/master/sample-output.pdf.

Updating / renewing the encryption key

Check out ttps://github.com/dirkx/gpg-offline-batch-key- and make sure you can run the 'sh gen.sh -d' as described in its README. This is to ensure you have all the dependencies (such as qrencode, libquirc and so on). This should complete with a generated PDF.

Next check that your printing subsystem works and that by default it goes to the right printer -- by doing:

     echo test | lpr

If not -edit gen.sh. Now run this script `for real' (ideally with a second person present; e.g. under Four Eye principles.

The result of this should be a file, e.g. ```public-key-45557.gpg``` that contains the public key. Copy this file into /etc/duplicity.

Check that you can read it there; and extract the key identifier:

Next import it into the local key store with:

     export GNUPGHOME=`pwd`
     gpg --import public-key-45557.gpg

This should show something such as

     gpg: key 534B2DDEB431D5B4: public key "key-45557" imported
     gpg: Total number processed: 1
     gpg:               imported: 1

Take note of the key id (534B2DDEB431D5B4) shown. Next make sure your GPG trusts it:

     gpg --edit-key 534B2DDEB431D5B4 trust quit

Select option 5 (ultimate) and confirm.

Then edit 'run.sh and add the key (534B2DDEB431D5B4) in this example to the list. It should be in the hidden encryption key list.

Finally run the scrip to check all is well. This will take a few minutes.

    sh run.sh
    sh run.sh incremental

Ransomware/targeted risk

This approach is not overly resistant against a targeted delete - as the sftp user can delete/modify files (as the retention is currently done from the 'source'). This is, to some extend, mitigated by zfs snapshots -- but not sufficiently at this time.

Restoring a file

To restore a single file - there is a script in

  /etc/duplicity/restore.sh

that is a wrapper for duplicity restore. Typical use is

   restore.sh restore --file-to-restore etc/foo/file.txt --time 2023-01-01

and it will put this in 'restored-file'.

History

2022/12 -- Changed to longer incremental runs; with only monthly full's.