Server backups / Duplicity

Backup

There is a regular backup ran from systemd; /etc/duplicity. It starts with a full dump of all MySQL tables. It is encrypted against a public key; the private key of which is held by the Trustees of the foundation.

Crontab kicks off a script;

  # monthly full, incrementals during the week.
  #
  3 3  1    * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh full
  3 3  2-31 * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh incremental

  # Half year retention for full; 1 months for the incrementals
  #
  1 1  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-but-n-full 6
  1 2  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-inc-of-but-n-full 1

The script: run.sh:

     #!/bin/sh
    set -e
    umask 077
    
    HOST="xxx.backup.host"
    
    DIR=/etc/duplicity
    W=incremental
    if [ $# != 0 ]; then
    	W=$1
    	shift
    fi
    T=
    if [ $W = full -o $W = incremental ];then
    	T=/
    	mysqldump --all-databases --single-transaction --quick --lock-tables=false  | gzip -9 > /var/lib/mysql-files/mysql-dump.gz
    fi
    
    # Verbose level 2 is errors and warnings; this way we skip
    # notices and quell all output if the backup is a success.
    #
    PASSPHRASE="YYYPASSWORD" \
         LANG=en_US.UTF8  LC_CTYPE=C HOME=$DIR GNUPGHOME=$DIR  \
         PYTHONWARNINGS="ignore::DeprecationWarning" \
    		python3 -W ignore::DeprecationWarning /usr/bin/duplicity $W $* \
    		\
    			-v 2 \
    			--hidden-encrypt-key XXXXX \
    			--hidden-encrypt-key YYYYY \
    			--sign-key           YYYYY \
    			--ssh-options="-i $DIR/backup.sftp -oUserKnownHostsFile=$DIR/knownhosts" \
    			--no-print-statistics \
    	\
    	--include /etc \
    	--include /usr/share/mediawiki \
    	--include /usr/share/wordpress \
    	--include /usr/local/makerspaceleiden-crm \
    	--exclude /var/lib/lxcfs \
    	--include /var/lib \
    	--include /var/www \
    	--include /var/log \
    	--exclude /dev \
    	--exclude /sys \
    	--exclude /run \
    	--exclude /tmp \
    	--exclude /snap \
    	--exclude /var/tmp \
    	--exclude /proc \
    	--exclude /swapfile \
    	--exclude /etc/duplicity/.cache \
    	\
    	$T \
    	sftp://msl@$HOST/backups 2>&1 |tee /var/log/last-duplcity-backup.new | grep -v DeprecationWarning | grep -v algorithm=hashes.SHA1
    	mv /var/log/last-duplcity-backup.new /var/log/duplicity.log
    	mv /var/log/duplicity.log.gz /var/log/duplicity.prevlog.gz || true
    	gzip /var/log/duplicity.log || true
    exit $?

Importing a new public key is done as follows

   cd /etc/duplicity
   cp XXXX/public-key-12345.gpg .
   gpg --homedir . --import public-key-12345.gpg .
   gpg --homedir . --edit-key XXXXXX
     trust 5
     save

And test by running it manually

  sudo /etc/duplicity/run.sh incremental

Safekeeping of private key

Example pgp key as QR

The code at https://github.com/dirkx/gpg-offline-batch-key- can be used to keep a backup off line.

It yields a printout like https://github.com/dirkx/gpg-offline-batch-key-/blob/master/sample-output.pdf.

Updating / renewing the encryption key

Check out ttps://github.com/dirkx/gpg-offline-batch-key- and make sure you can run the 'sh gen.sh -d' as described in its README. This is to ensure you have all the dependencies (such as qrencode, libquirc and so on). This should complete with a generated PDF.

Next check that your printing subsystem works and that by default it goes to the right printer -- by doing:

     echo test | lpr

If not -edit gen.sh. Now run this script `for real' (ideally with a second person present; e.g. under Four Eye principles.

The result of this should be a file, e.g. ```public-key-45557.gpg``` that contains the public key. Copy this file into /etc/duplicity.

Check that you can read it there; and extract the key identifier:

     export GNUPGHOME=`pwd`
     gpg2 < public-key-45557.gpg

This should show something like

     pub   ed25519 2023-10-13 [SC]
          18109A9D225C92999511E565E399BDCDE55FFF07
     uid           key-45557
     sub   cv25519 2023-10-13 [E]

Next import it into the local key store with:

     export GNUPGHOME=`pwd`
     gpg2 --import public-key-45557.gpg

And make sure your GPG trusts it:

     gpg --edit-key 18109A9D225C92999511E565E399BDCDE55FFF07 trust quit

Then edit 'run.sh and add the key (18109A9D225C92999511E565E399BDCDE55FFF07) in this example to the list.

Finally run the scrip to check all is well. This will take a few minutes.

    sh run.sh
    sh run.sh incremental

Ransomware/targeted risk

This approach is not overly resistant against a targeted delete - as the sftp user can delete/modify files (as the retention is currently done from the 'source'). This is, to some extend, mitigated by zfs snapshots -- but not sufficiently at this time.

Restoring a file

To restore a single file - there is a script in

  /etc/duplicity/restore.sh

that is a wrapper for duplicity restore. Typical use is

   restore.sh restore --file-to-restore etc/foo/file.txt --time 2023-01-01

and it will put this in 'restored-file'.

History

2022/12 -- Changed to longer incremental runs; with only monthly full's.