Server backups / Duplicity

Backup

There is a regular backup ran from systemd; /etc/duplicity. It starts with a full dump of all MySQL tables. It is encrypted against a public key; the private key of which is held by the Trustees of the foundation.

Initial installation

   apt install duplicity

Crontab kicks off a script;

  # monthly full, incrementals during the week.
  #
  3 3  1    * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh full
  3 3  2-31 * *	root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh incremental

  # Half year retention for full; 1 months for the incrementals
  #
  1 1  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-but-n-full 6
  1 2  * * 1	 root test -x  /etc/duplicity/run.sh && /etc/duplicity/run.sh remove-all-inc-of-but-n-full 1

The script: run.sh:

     #!/bin/sh
    set -e
    umask 077
    
    HOST="xxx.backup.host"
    
    DIR=/etc/duplicity
    W=incremental
    if [ $# != 0 ]; then
    	W=$1
    	shift
    fi
    T=
    if [ $W = full -o $W = incremental ];then
    	T=/
    	mysqldump --all-databases --single-transaction --quick --lock-tables=false  | gzip -9 > /var/lib/mysql-files/mysql-dump.gz
    fi
    
    # Verbose level 2 is errors and warnings; this way we skip
    # notices and quell all output if the backup is a success.
    #
    PASSPHRASE="YYYPASSWORD" \
         LANG=en_US.UTF8  LC_CTYPE=C HOME=$DIR GNUPGHOME=$DIR  \
         PYTHONWARNINGS="ignore::DeprecationWarning" \
    		python3 -W ignore::DeprecationWarning /usr/bin/duplicity $W $* \
    		\
    			-v 2 \
    			--hidden-encrypt-key XXXXX \
    			--hidden-encrypt-key YYYYY \
    			--sign-key           YYYYY \
    			--ssh-options="-i $DIR/backup.sftp -oUserKnownHostsFile=$DIR/knownhosts" \
    			--no-print-statistics \
    	\
    	--include /etc \
    	--include /usr/share/mediawiki \
    	--include /usr/share/wordpress \
    	--include /usr/local/makerspaceleiden-crm \
    	--exclude /var/lib/lxcfs \
    	--include /var/lib \
    	--include /var/www \
    	--include /var/log \
    	--exclude /dev \
    	--exclude /sys \
    	--exclude /run \
    	--exclude /tmp \
    	--exclude /snap \
    	--exclude /var/tmp \
    	--exclude /proc \
    	--exclude /swapfile \
    	--exclude /etc/duplicity/.cache \
    	\
    	$T \
    	sftp://msl@$HOST/backups 2>&1 |tee /var/log/last-duplcity-backup.new | grep -v DeprecationWarning | grep -v algorithm=hashes.SHA1
    	mv /var/log/last-duplcity-backup.new /var/log/duplicity.log
    	mv /var/log/duplicity.log.gz /var/log/duplicity.prevlog.gz || true
    	gzip /var/log/duplicity.log || true
    exit $?

Importing a new public key is done as follows

   cd /etc/duplicity
   cp XXXX/public-key-12345.gpg .
   gpg --homedir . --import public-key-12345.gpg .
   gpg --homedir . --edit-key XXXXXX
     trust 5
     save

And test by running it manually

  sudo /etc/duplicity/run.sh incremental

Safekeeping of private key

The code at https://github.com/dirkx/gpg-offline-batch-key- can be used to keep a backup off line. It yields a printout like https://github.com/dirkx/gpg-offline-batch-key-/blob/master/sample-output.pdf. This makes sure that there are no `on line' copies of that private key; just one off-line; on paper.

Note that the backup process itself is also privileged (i.e. it can access anything on the machine) and thus has access to any and all data on the machine.

Example pgp key as QR

Ransomware/targeted risk

This approach is not overly resistant against a targeted delete - as the sftp user can delete/modify files (as the retention is currently done from the 'source'). This is, to some extend, mitigated by zfs snapshots -- but not sufficiently at this time.

Restoring a file

To restore a single file - there is a script in

  /etc/duplicity/restore.sh

that is a wrapper for duplicity restore. Typical use is

   restore.sh restore --file-to-restore etc/foo/file.txt --time 2023-01-01

and it will put this in 'restored-file'.

Updating / renewing the encryption key

Check out ttps://github.com/dirkx/gpg-offline-batch-key- and make sure you can run the 'sh gen.sh -d' as described in its README. This is to ensure you have all the dependencies (such as qrencode, libquirc and so on). This should complete with a generated PDF.

Next check that your printing subsystem works and that by default it goes to the right printer -- by doing:

     echo test | lpr

If not -edit gen.sh. Now run this script `for real' (ideally with a second person present; e.g. under Four Eye principles.

The result of this should be a file, e.g. ```public-key-45557.gpg``` that contains the public key. And a printout - that contains the private key.

Once the script completes - the ONLy copy of the private key is in that paper document. It is not present anymore in digital form. The only thing left behind digital is the public key. I.e. the file which you can use to _encrypt_; but which cannot _decrypt_.

Copy the public key file into /etc/duplicity. Check that you can import it there; and extract the key identifier:

     export GNUPGHOME=`pwd`
     gpg --import public-key-45557.gpg

This should show something such as

     gpg: key 534B2DDEB431D5B4: public key "key-45557" imported
     gpg: Total number processed: 1
     gpg:               imported: 1

Take note of the key id (534B2DDEB431D5B4) shown. Next make sure your GPG trusts it:

     gpg --edit-key 534B2DDEB431D5B4 trust quit

Select option 5 (ultimate) and confirm.

Then edit 'run.sh and add the key (534B2DDEB431D5B4) in this example to the list. It should be in the hidden encryption key list -- i.e where it says XXXX in the above example.

Finally run the scrip to check all is well. This will take a few minutes.

    sh run.sh
    sh run.sh incremental

History

2022/12 -- Changed to longer incremental runs; with only monthly full's.