Website mijn.makerspaceleiden.nl setup

On a demo/local laptop

git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git

   cd makerspaceleiden-crm
   sh loaddemo.sh

then visit <a href="http://localhost:8000/">http://localhost:8000/</a>.

(edit two things in requirements.txt. First, the latest version of Django is 4.1.2 This version causes errors. So take an older version of Django. Second thing is import python-dateutil).

So ```requirements.txt``` starts with:

    Django==3.2.10
    python-dateutil
    pytz
    wheel
    django-import-export
    etc...

Note that going to version 4 is fairly trivial with ```django-upgrade``` - but requires a fix in one dependency (https://github.com/linevych/django-search-admin-autocomplete/issues/15). Perhaps we should remove that dependency - it is only used by the trustees really.

Prepare for restarts when editing code with:

source venv/bin/activate

and then do :

python3 manage.py runserver

as needed (most changes will be picked up automatically).

On the production server -- how to update

Check that you are in the 'crmadmin' group. Then:

 cd /usr/local/makerspaceleiden-crm
 git pull <right version and branch>

Activate the right vertual environment (rebuild with python3 -menv venv):

 source venv/bin/activate

If needed - make a backup of the database with either

 python3 manage.py dumpdata

or (passowrd in makerspaceleiden/my.cnf):

 mysqldump -u mslcrmuser  -p mslcrm

Check for any new static stuff & updates:

 pip3 install -r requirements.txt 

Next migrate the database if needed:

 python3 manage.py makemigrations
 python3 manage.py migrate

If the asset changes - then also run the collect static:

 python3 manage.py collectstatic --dry-run --noinput

examine the output and then either rerun the command or manually adjust.

And finally restart the webserver:

  sudo apachectl restart

NOTE: var/media contains the images.

So while you can actually wack & redo the entire setup - you will need to preserve var/media. (Perhaps we should move var/media to /var/media - fully outside the tree - as it is also the only thing that 'www-data' can write to.

In production - setup from zero

Note: pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).

Make sure the baseline tools are present:

  sudo apt-get install python3 libmysqlclient  default-libmysqlclient-dev msmtp-mta apache2 libapache2-mod-uwsgi

The default-libmysqlclient-dev module is needed as pip3 wants mysqlconfig - which is not in the baseline libmysqlclient.

Initial checkout of code:

   cd /usr/local
   git clone https://github.com/dirkx/makerspaceleiden-crm.git

Make evertyin group owned (crmadmin) and add that group to the accounts of those that need to maintain it.

Create random seed

   openssl rand 128 > /etc/crm_secret_key.txt
   chmod 640 /etc/crm_secret_key.txt
   chgrp www-data /etc/crm_secret_key.txt
   

Allow storing of uploads in media by the suid that the webserver runs as:

   mkdir -p var/media
   chown www-data var/media
   

Allow server to rotate/recreate logfiles on the fly

   mkdir /var/log/crm
   chown www-data:crmadmin /var/log/crm
   chmod 770 /var/log/crm

Prepare env for python and pull in the various dependencies.

   python3 -mvenv venv
   source ./venv/bin/activate
   pip3 install -r requirements
   

Set up the framework

   cd makerspaceleiden
   ln -s prod.py local.py
   cat > makerspaceledien/my.cnf <<EOM
   [client]
   database = mslcrm
   user = mslcrmuser
   password = XXXX-passowrd-XXX
   default-character-set = utf8
   EOM
   

Create database & user

   mysql (suply database admin arguments as and when needed)
   create database mslcrm;
   create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
   grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
   flush priveleges;
   

Check for issues, init and build database & site

   python3 manage.py check --deploy
   python3 manage.py makemigrations
   python3 manage.py migrate
   python3 manage.py collectstatic
   

Check that email works:

  python3 manage.py sendtestemail your@email.address.com

Create temp super user so you can log into complete the setup

   python3 manage.py createsuperuser
   

Create apache config

   cat > /etc/apache2/sites-available/crm.conf <<EOM
   WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
   
   WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
   WSGIProcessGroup  crm 
   
   Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
   Alias /media/      /usr/local/makerspaceleiden-crm/var/media/
   
   <Directory /usr/local/makerspaceleiden-crm/>
     <Files wsgi.py>
        Require all granted
     </Files>
   </Directory>
   
   <Directory /usr/local/makerspaceleiden-crm/static>
           Options None
           order deny,allow
           allow from all
           Require all granted
   </Directory>
   
   <Directory /usr/local/makerspaceleiden-crm/var/media>
           Options None
           order deny,allow
           allow from all
           Require all granted
   </Directory>
   EOM

Activate this setup

   ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf 
   

Start server and keep an eye on the log:

   apacheclt configtest
   apachectl restart
   tail -F /var/log/apache2/error.log

Now go to https://makerspaceleiden.nl/crm and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.

Setup Signal Bridge

Requires openjdk, java-dbus-bin and dbussy. Binary package from Shttps://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).

Post install - as user signal-cli - activation is required:

  $ signal-cli -u +317... register -voice
  $ signal-cli -u +317... verify code from call

which will create the right structure in ~/.local.

Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.

Logfile retention

Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.

See https://github.com/dirkx/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.

Cleanup and reminder crons

To be configured/written

Backup

This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database and captures the whole directory -- which includes var/media -- the uploads.

See the section backup on the general Server setup page.

Mailing list integration

Mailing lists are ran by a third party - we currently use a hack to manage these by faking browser access to their form API.

experimental update

Integration via their newly fangled "Sympa" web service.

Security/Privacy consideration

Our mailing list provider is a processed of our data; and distributes the email to our members. This is the basis for allowing this party access to the email addresses or our users. We can limit the information to exactly that.

Additional measures:

1. Gather the /just/ needed data in a single purpose view
2. Specific user that is restricted to just that view and has only select/view; no write.
3. Strong password & secure transfer of the password to the third party
4. Require the use of SSL/TLS for this user in mysql
5. Lock down of this user to a specific IP in mysql
6. Use of IP tables to lock down the connection to just the
7. Move to a non-common port

Still open:

1. Does the API still need the (full) name - as mailman currently does - for use in the From: ? Or can we further limit this.
2. Can we use X.509 client certs for authentication.

configuration

Create a mysql user specific to sympa; $SYMPA_IP is the IP address of their gateway (currently just one) and $secret is a strong password:

    create user mlist@$SYMPA_IP identified by '$secret' REQUIRE SSL;

Then create a view with just the data needed:

    create view mailinglists_api as
       select distinct concat(first_name,' ',last_name) "name",name "list", email
               from members_user,mailinglists_subscription,mailinglists_mailinglist
       where   mailinglists_subscription.member_id = members_user.id and
               mailinglists_subscription.mailinglist_id = mailinglists_mailinglist.id and
               mailinglists_subscription.active =1
       order by email
       with read only;

And grant this integration user access to just that view:

    GRANT SELECT, SHOW VIEW ON mslcrm.mailinglists_api TO mlist@ $SYMPA_IP;

Then provide the mailinglist provider with the following file/data:

   db_type mysql
   db_port $PORT 
   db_host $OUR_IP
   db_user mlist
   db_passwd $secret
   db_name mslcrm
   sql_query select distinct name,email from mailinglists_api where list = 'spacelog';

Configure SSL on the right port

   [mysqld]
   ssl_ca=/etc/letsencrypt/live/makerspaceleiden.nl/chain.pem
   ssl_cert=/etc/letsencrypt/live/makerspaceleiden.nl/fullchain.pem
   ssl_key=/etc/letsencrypt/live/makerspaceleiden.nl/privkey.pem
   require_secure_transport=ON
   port=$PORT

And allow acccess:

    sudo ufw allow in from $SYMPA_IP to any port $PORT

or

   iptables -A INPUT  -p tcp -s $SYMPA_IP --sport 1024:65535 -d $OUR_IP --dport $PORT \
          -m state --state NEW,ESTABLISHED -j ACCEPT
   iptables -A OUTPUT -p tcp -s 2$OUR_IP --sport $PORT -d $SYMPA_IP --dport 1024:65535 \
          -m state --state ESTABLISHED -j ACCEPT