Server setup

The front end server (mid 2018) runs on Linux; in a `cloud' hosted environment.

Below documents the initial setup of the base machine; followed by the setup for each of the modules.

The final section shows the monthly and annual maintenance cycles.

Setup and rudimentary hardening

• Get the machine in a known state and install sudo (so we can disable root; and comply with 'named accounts' only policies):

apt update
apt upgrade
apt install sudo certbot certbot-apache moreutils

• enable ufw and allow the usual ports in IPv4 and IPv6

 for port in 22 25 53 80 443 1883
 do
   ufw allow  $port
 done
 # do this last so we're not kicked out.
 ufw enable

• create named accounts for each of the admins (you need to get everyones their public SSH key):

 adduser \
  --system \
  --shell /bin/bash \
  --gecos 'Dirk-Willem van Gulik' \
  --group \
  --ingroup admin \
  --disabled-password \
  dirkx

• Add an ssh key for each of these users

• check that you can log in; and sudo with at least one of them.

• Block root login and passwords in /etc/ssh/sshd.conf:

 PermitRootLogin no
 PasswordAuthentication no
 ChallengeResponseAuthentication no

Note: if you did not check the sudo/login of an admin user - they you are about to lock yourself out upon reboot.

• Edit /etc/sysctl.conf to block spoofing, ICMP broadcast, source-packet routing, send redirect, SYN attacks, Martians and ICM redirects.

• Prevent IP spoofing for DNS by replacing multi on to nospoof on in /etc/hosts.conf

• Securing shared memory.

 echo tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0 >> /etc/fstab

• Make sure we trap all crontab outputs systemctl edit cron.service:

  [Service]
  Environment="MAILTO=noc@makerspaceleiden.nl"

• Reboot.

MQTT Server install

Installed with:

  sudo ufw allow 1883
  sudo apt-get update
  sudo apt-get install mosquitto

Enable on boot and start with

   sudo systemctl enable mosquitto 
   sudo systemctl start mosquitto

check with

  systemctl status mosquitto 

Then secure with

  ....

Setup of the basic website

We need to first set up a very basic website; in order to be able to fetch the required SSL certificates.

• Install apache and certbot and the integration glue between the two:

sudo apt install apache2 certbot python-certbot-apache

• Request the needed certs:

sudo certbot --apache -d makerspaceleiden.nl -d www.makerspaceleiden.nl -d wiki.makerspaceleiden.nl

• Ensure they get renewed; and that the admins are emailed when this goes wrong:

Setup of the MTA

Log in as one of the admins.

• Make sure /etc/mailname is set to makerspaceleiden.nl

• Install postfix and basic mail stuff:

 sudo apt install postfix mailtools postgrey mavisd-new spamassassin clamav-daemon libnet-dns-perl libmail-spf-perl pyzor razor  arj bzip2 cabextract cpio file gzip  nomarch pax rar unrar unzip zip postsrsd

• Edit the /etc/aliases file to redirect the mail of 'root' and update:

 sudo vi /etc/aliases
 sudo newaliases

• Edit the certs in /etc/postfix/main.cf

 mydomain=makerspaceleiden.nl # needed for SRS
 smtpd_tls_cert_file=/etc/letsencrypt/live/makerspaceleiden.nl/fullchain.pem
 smtpd_tls_key_file=/etc/letsencrypt/live/makerspaceleiden.nl/privkey.pem

 ... at the end add ...
 # SPF rewrite
 sender_canonical_maps = tcp:localhost:10001
 sender_canonical_classes = envelope_sender
 recipient_canonical_maps = tcp:localhost:10002
 recipient_canonical_classes= envelope_recipient,header_recipient

and check that your apt-install has created an /etc/postsrd.secret file that is root only. If not created it with a few 100 bytes of randomness.

• Make sure there is a postfix restarted in /etc/letsencrypt/renewal-hooks/post. As otherwise postfix won't see your fresh certs. A shell script with just something like below should do the trick:

  #!/bin/sh
  service postfix restart

• Give the virus/spam scanner mutual access:

 sudo adduser clamav amavis
 sudo adduser amavis clamav

• As the amavis user; razor-admin -create and razor-admin -register if needed.

• Edit /etc/amavis/conf.d/15-content_filter_mode to activate the filters and restart:

 sudo /etc/init.d/amavis restart

• The whitelist for postgray contains a number of entries for the various deur logging systems:

... to do ...

Setup of Apache

   sudo apt install apache2 letsencrypt

Configure let's encrypt and add the flag reuse-key to CLI.ini. This is to allow the payment system to keep trusting the server even during let's encrypt updates every 90 days.

Setup of WordPress

• Install enough of the LAMP stack to get going:

 sudo apt install mysql-server php libapache2-mod-php php-mysql wordpress-theme-twentyseventeen wordpress

• Secure your mysql install:

 sudo mysql_secure_installation

• Configure apache:

  cat <<EOM > /etc/apache2/sites-available/wordpress.conf
 <VirtualHost *:80>
     ServerAdmin noc@makerspaceleiden.nl
  
     ServerName        makerspaceleiden.nl
     ServerAlias       www.makerspaceleiden.nl
  
     DocumentRoot /var/www/html
     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined
  
     RewriteEngine On
     RewriteCond %{HTTPS} off
     RewriteRule (.*) https://%{SERVER_NAME}/$1 [R,L]
 </VirtualHost>
 
 Listen *443
 <VirtualHost *:443>
     ServerAdmin noc@makerspaceleiden.nl
     ServerName        makerspaceleiden.nl
     ServerAlias       www.makerspaceleiden.nl
  
     SSLCertificateFile  /etc/letsencrypt/live/makerspaceleiden.nl/fullchain.pem
     SSLCertificateKeyFile /etc/letsencrypt/live/makerspaceleiden.nl/privkey.pem
 
     SSLProtocol all -SSLv2 -SSLv3
     SSLHonorCipherOrder On
     SSLCipherSuite EECDH+AES:EDH+AES:-SHA1:EECDH+RC4:EDH+RC4:RC4-SHA:EECDH+AES256:EDH+AES256:AES256-SHA:!aNULL:!eNULL:!EXP:!LOW:!MD5
 
     DocumentRoot /usr/share/wordpress
 
     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined
  
     <Directory /usr/share/wordpress>
           Options FollowSymLinks
           AllowOverride Limit Options FileInfo
           DirectoryIndex index.php
           Order allow,deny
           Allow from all
     </Directory>
 
     <Directory /usr/share/wordpress/wp-content>
           Options FollowSymLinks
           Order allow,deny
           Allow from all
       </Directory>
        
  </VirtualHost>

  EOM

• Create a database and a config file (changing yourpasswordhere123):

 cat <<EOM |  sudo mysql --defaults-extra-file=/etc/mysql/debian.cnf
 CREATE DATABASE wordpress;
 GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER
 ON wordpress.*
 TO wordpress@localhost
 IDENTIFIED BY 'yourpasswordhere123';
 FLUSH PRIVILEGES;
 EOM

 cat <<EOM > /etc/wordpress/config-localhost.php
 <?php
 define('DB_NAME', 'wordpress');
 define('DB_USER', 'wordpress');
 define('DB_PASSWORD', 'yourpasswordhere123');
 define('DB_HOST', 'localhost');
 define('WP_CONTENT_DIR', '/usr/share/wordpress/wp-content');
 define('FS_METHOD', 'direct');
 ?>

• Secure this file so that only the webserver can see this password.

    sudo chown root:www-data /etc/wordpress/config-localhost.php
    sudo chmod o-rwx,g-wx /etc/wordpress/config-localhost.php

• Enable, kill default and restart:

  sudo a2ensite wordpress
  sudo a2dissite 000-default 
  sudo systemctl reload apache2

• Check that it all works by visiting

  httpp://makerspaceleiden.nl/install.php

• If you want to be able to update in place - be sure to have the wordpress content directory `www-data' owned.

Updaten van Word press

Zie Wordpress Update

Setup of Media Wiki

• Install the base packages

 apt install mediawiki imagemagick php-apcu php-intl

• Create the database:

cat <<EOM |  sudo mysql --defaults-extra-file=/etc/mysql/debian.cnf
CREATE DATABASE mediawiki;
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX
ON mediawiki.*
TO mediawiki@localhost
IDENTIFIED BY 'yourpasswordhere12443';
FLUSH PRIVILEGES;
EOM

• Restart apache.

• Go to the wiki - and use the config values from above DB setup.,

• Follow the instructions and copy the generated LocalSettings.php to the specified location.

• Add the various users.

making a backup and importing it

• Backup script:

 #!/bin/sh
 set -e
 set -x
 D=`date +%Y%m%d%H%M%S`
 mkdir media-wiki-backup.$D
 
 cd media-wiki-backup.$D
 
 mysqldump --user=wikiuser --password="XXXX" wikidb > file.sql
 mysqldump --user=wikiuser --password="XXXX" wikidb --xml > file.xml
 
 cp -r /usr/local/www/mediawiki/images .
 cp /usr/local/www/mediawiki/LocalSettings.php .
 
 cd ..
 tar zcf media-wiki-backup-$D.tgz media-wiki-backup.$D
 rm -rf  media-wiki-backup.$D

• Dump the full wiki:

    jexec mls  ... script ..

• Copy the file across.

• Import it on the other machine

  #!/bin/sh
  set -ex
  
  if ! test -f latest.tgz; then
   		echo no last dump.
   		exit 1
   fi
   mkdir tmp.$$
   cd tmp.$$
   tar zxf ../latest.tgz
   (
   cd *
   
   rm -rf /var/lib/mediawiki/cache /var/lib/mediawiki/images
   
   mkdir import /var/lib/mediawiki/cache /var/lib/mediawiki/images
   cp -r images/? /var/lib/mediawiki/images
   chown -R www-data:www-data /var/lib/mediawiki/images  /var/lib/mediawiki/cache
   
   cat <<EOM |  sudo mysql --defaults-extra-file=/etc/mysql/debian.cnf
   DROP DATABASE mediawiki;
   CREATE DATABASE mediawiki;
   GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX
   ON mediawiki.*
   TO mediawiki@localhost
   IDENTIFIED BY 'XXXX';
   FLUSH PRIVILEGES;
   EOM
  
   cat file.sql | sudo mysql --defaults-extra-file=/etc/mysql/debian.cnf mediawiki
  
   sudo -u www-data /usr/share/mediawiki/maintenance/update.php
   sudo -u www-data php  /usr/share/mediawiki/maintenance/cleanupImages.php
   sudo -u www-data php /usr/share/mediawiki/maintenance/rebuildall.php 
   sudo -u www-data php /usr/share/mediawiki/maintenance/rebuildImages.php 
   
   ) || exit 1
   rm -rf tmp.$$
   exit $?

Monthly Reporting of deelnemers count

The provider has set up a crontab (through their support desk, manual action) that emails a uu-encoded gzipped lists of subscribers to all list owners.

• The user 'listreporter' has been created as an alias in /etc/aliases

 listreporter: |/etc/mailman/listreporter.sh

• and is activated with newaliases.

This is wired up to a swapping script (requiring apt install moretools) and a script that is somewhat paranoid and parses/unpacks the email:

 #!/bin/sh
 set -e
 umask 077
 test -x /etc/mailman/listreporter.pl || exit 0
 TMPDIR=${TMPDIR:-/tmp}
 TMPDIR=${TMPDIR}/mailmanlistreporter.$$
 (
 	mkdir $TMPDIR
 	cd $TMPDIR
       # limit sizeto prevent naughtyness
 	dd bs=1k count=32 -out=in.msg
 	/usr/bin/perl /etc/mailman/listreporter.pl
  ) 2>&1 | ifne mail -s "Fault with the list reporter." noc@makerspaceleiden.nl
  rm -rf "${TMPDIR}"
  exit 0

The somewhat unusual ifne is from moretools and ensures that mail only runs if there is actual stdin text; so you do not get empty emails if all is well. This wrapper then wired to the script sending out the actual email:

  #!/usr/bin/perl 
  # use strict;
  use IO::CaptureOutput qw/capture_exec/;
  
  umask(077);
  
  open(STDIN,"in.msg") or die "Could not open msg file: $!\n";
  
  my $list;
  my $subject;
  
  while(<STDIN>) {
          last if m/^\s*$/;
          $subject = $1 if m/^Subject\s*:\s*(.*)$/;
  };
  close(STDIN);
  
  open STDIN, '</dev/null'; # Prevent warnings deep down in IO capture.
  
  $list = $1 if $subject =~ m/List subscriber file:\s*(\S+)/;
  
  # exit 0 unless $list;
  
  die "Not the right subject <$subject>\n" unless $list;
  die "Not a list I manage." unless $list eq 'deelnemers';
  
  my ($stdout, $stderr, $success, $exit_code) = capture_exec("munpack","-q","in.msg");
  
  die "Unpacking failed: $!\n" unless $exit_code == 0;
  
  my $d = ;
  foreach my $f (<subscribers.$list.*.gz>) {
          next unless $f =~ m/subscribers.$list.([0-9\-]+).txt.gz/;
          $d = $1;
  
          system("cat '$f' | gunzip -c | dd bs=1k count=32 of=list.txt") == 0
                  or die "Could not unpack $f\n";
          open(FH,'list.txt') or die "Could not open subscriber list";
          my @list = ();
          while(<FH>) {
                  chomp;
                  s/\s*#.*//;
                  s/\P{IsPrint}//g;
                  next if m/^\s*$/;
                  push @list, $_;
          };
          my $count= @list;
  
          open(FH,"| mail -s 'Makerspace Leiden, $count $list on $d' -aFrom:hetbestuur\@makerspaceleiden.nl  $list\@makerspaceleiden.nl")
                  or die "Cannot open pipe.\n";
  
          print FH <<"EOM";
  Hello All,
  
  We have currently $count makers united, listed below:
  
  Your can edit your own information here:
  
          https://mailman.makerspaceleiden.nl/mailman/options/deelnemers
  
  and this is also where you can switch to a digtest (just one big message/day)
  or see the archive of historic messages.
  
  EOM
          print FH join("\n", @list);
          print FH "\n-- \nStiching Makerspace Leiden / hetbestuur\@makerspaceleiden.nl\n\n";
          close(FH);
  
          exit(0);
  }

this is then matched by sufficient permission to allow this post to go through moderation unchecked. We should get an email from the hoster on the first of every month.

MQTT Monitoring

A simple perl script listens to the MQTT bus; and flags up any node that has not reported in in a while.

  #!/usr/bin/perl
  use strict;
  use Email::MIME;
  use Email::Sender::Simple qw(sendmail);
  
  $|++;
  
  my $TO = 'deelnemers@makerspaceleiden.nl';
  my $reporting_period = 600;
  
  # The XDG_CONFIG_HOME is to prevent a warning about the Euid not having a home dir.
  open(STDIN,"XDG_CONFIG_HOME=/tmp /usr/bin/mosquitto_sub -v -h space.makerspaceleiden.nl -t 'log/#' |") or die $@;
  
  # We rotate the logs - as to be able to compress and retire them; thus preventing a full
  # disk. We could also modify this - and rely on systemd to do such for us.
  #
  # We use a sub-directory of 'log' as to allow us to be ran as `nobody'. That is a bit
  # safer, especially as we get 'data from the raw internet'
  #
  open(STDOUT,"|/usr/bin/rotatelogs -lf /var/log/mqtt/doorlog.%Y.%m.%d 86400") or die $@;
  
  my %lastseen = ();
  my %reported = ();
  
  # Take a list of items; and return them with a 'and' between the one but last 
  # and last one. Just to make things look a bit nicer during reporting.
  #
  sub nice_join {
      return "none" if !@_;
      my $last = pop;
      return $last if !@_;
      return join(', ', @_) . " and $last";
  }
  
  $SIG{ALRM} = sub {
  	my @lost = ();
  
  	map {
  		push @lost, $_
  			if time() - $lastseen{ $_ } > $reporting_period *2;
  	} keys %lastseen;
  
  	if (@lost) {
  		my @nope = ();
  		map { 
  			if ($reported{ $_ } ++ > 3) {
  				delete $reported{ $_ };
  				delete $lastseen{ $_ };
  				push @nope, $_;
  			};
  		} @lost;
  		my $subject = sprintf("The connection%s to doornode%s %s %s down.", 
  				(@lost > 1) ? "s" : "", 
  				(@lost > 1) ? "s" : "", 
  				nice_join(@lost),
  				(@lost > 1) ? "are" : "is"
  			);
  		my $ps = ;
  		$ps = "PS: I won't be reporting on ".nice_join(@nope)." any more - down too long.\n"
  			if (@nope);
  
  		my $message = Email::MIME->create(
  		  header_str => [
  		    From    => 'Monitoring MSL <noc@makerspaceleiden.nl>',
  		    To      => $TO,
  		    Subject => $subject
  		  ],
  		  attributes => {
  		    encoding => 'quoted-printable',
  		    charset  => 'ISO-8859-1',
  		  },
  		  body_str => "$subject If you are at the makerspace - please be so kind run a reset.\n".
  		  	"\n".
  		  	"See the table at:\n".
  		  	"\n".
  			"    https://wiki.makerspaceleiden.nl/mediawiki/index.php/NetworkSetup2018\n".
  		  	"\n".
  			"for the right port/cable to unseat for 5 seconds and plug back in. That should ".
  			"fix it. If you do so - let the mailing list know.\n".
  			"\n".
  			"As the doornodes remember recent keys - you may still be able to open the door, ".
  			"especially if you entered in the last week or so. But then again - perhaps not.\n".
  			"\n".
  			"Alternatively Aart, Hans and Dirk have physical spare keys. Dirk's key can be ".
  			"picked up during officehours at the janvossensteeg.\n".
  			"\n".
  			"Thanks !\n\n$ps\n"
  		);
  
  		sendmail($message);
  
  	};
  
  	alarm $reporting_period;
  };
  alarm $reporting_period;
  
  while(<STDIN>) {
 	# We're on a raw bus - so do a modicum of filtering.
 	# 
 	s/[^[:ascii:]]//g;   	print;
   
  	next unless m/\[(\S+?)\]/;
  
  	$lastseen{$1} = time();
  }
  
  die "Lost MQTT connection.\n";

This script is tarted/managed by a trivial startup file.

  # /etc/systemd/system/mslmon.service 
  [Unit]
  Description=MSL Door node montitoring on MQTT
  After=network.target

  [Service]
  Type=simple
  User=nobody
  Group=nogroup
  WorkingDirectory=/tmp
  ExecStart=/home/dirkx/check-nodes.pl
  Restart=on-failure # or always, on-abort, etc

  [Install]
  WantedBy=multi-user.target

Manage with the usual systemd commands such as

   $ systemctl status mslmon.service
   ● mslmon.service - MSL Door node montitoring on MQTT
     Loaded: loaded (/etc/systemd/system/mslmon.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2018-07-23 17:03:46 CEST; 8min ago
   Main PID: 31903 (check-nodes.pl)
      Tasks: 4 (limit: 4583)
  CGroup: /system.slice/mslmon.service
          ├─31903 /usr/bin/perl /home/dirkx/check-nodes.pl
          ├─31914 sh -c /usr/bin/mosquitto_sub -v -h space.makerspaceleiden.nl -t 'log/#'
          ├─31918 /usr/bin/rotatelogs -lf /var/log/mqtt/doorlog.%Y.%m.%d 86400
          └─31919 /usr/bin/mosquitto_sub -v -h space.makerspaceleiden.nl -t log/#

And check the logs with

   $ sudo journalctl -u mslmon.service
   -- Logs begin at Sat 2018-06-23 13:19:36 CEST, end at Mon 2018-07-23 17:13:52 CEST. --
   Jul 23 17:03:46 makerspaceleiden.nl systemd[1]: Started MSL Door node montitoring on MQTT.

Finally - there is a crontab that purges things after 90 days:

  # Retain only the last 90 days.
  4 4 * * *       root    test -d /var/log/mqtt && ( ls /var/log/mqtt/doorlog.* | sort -n | tail +90 | while read f; do rm "$f"; done )

Obviously - this could be improved by having journal/systemd take care of this. And the script could be more clever about telling that a node came back up or taking into account the loss of the ADSL or MQTT brokers. And distingsuish these as separate failure modes.

Backup

Server backups / Duplicity

Ransomware/targeted risk

This approach is not overly resistant against a targeted delete - as the sftp user can delete/modify files (as the retention is currently done from the 'source'). This is, to some extend, mitigated by snapshots -- but not sufficiently at this time.

Virus scan

There is a malware/virus scan set up across the uploads; with automatic refresh of the signature DBs:

... todo

DKIM setup

Packages and perms:

 sudo apt-get install opendkim opendkim-tools postfix-policyd-spf-python postfix-pcre
 sudo adduser postfix opendkim

Add to /etc/postfix/master.cf

 policyd-spf  unix  -       n       n       -       0       spawn
     user=policyd-spf argv=/usr/bin/policyd-spf

And add to main.cf its smtpd_recipients_restrictions a

 check_policy_service unix:private/policyd-spf

Add/edit /etc/opendkim.conf as needed and force the right perms:

  chmod u=rw,go=r /etc/opendkim.conf
  mkdir /etc/opendkim
  mkdir /etc/opendkim/keys
  chown -R opendkim:opendkim /etc/opendkim
  chmod go-rw /etc/opendkim/keys

and generate the keys.

PAS OP: the generated DNS stub is wrong - it needs to have the 'rsa-' prefix removed from the bash. Know bug from 2017 https://github.com/linode/docs/commit/9464cbd43947c6e5de021f40c93828b410b44d8d not yet fixed.

Add to postfix its mail

 # Milter configuration
 # OpenDKIM
 milter_default_action = accept
 # Postfix ≥ 2.6 milter_protocol = 6, Postfix ≤ 2.5 milter_protocol = 2
 milter_protocol = 6
 smtpd_milters = local:opendkim/opendkim.sock
 non_smtpd_milters = local:opendkim/opendkim.sock

Tunnel for the kWh meter

See KWh meter setup.