Brontekst bekijken van Website mijn.makerspaceleiden.nl setup

[[Category:Procedures]][[Category:Docs]]
<h2>On a demo/local laptop</h2>    git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git
    cd makerspaceleiden-crm
    sh loaddemo.sh

then visit <a href="http://localhost:8000/">http://localhost:8000/</a>.


(edit two things in requirements.txt. First, the latest version of Django is 4.1.2 This version causes errors. So take an older version of Django. Second thing is import python-dateutil).

So ```requirements.txt``` starts with:

     Django==3.2.10
     python-dateutil
     pytz
     wheel
     django-import-export
     etc...

Note that going to version 4 is fairly trivial with ```django-upgrade``` - but requires a fix in one dependency (https://github.com/linevych/django-search-admin-autocomplete/issues/15). Perhaps we should remove that dependency - it is only used by the trustees really.

Prepare for restarts when editing code with:

 source venv/bin/activate

and then do :

 python3 manage.py runserver

as needed (most changes will be picked up automatically).

== On the production server -- how to update ==

Check that you are in the 'crmadmin' group. Then:

  cd /usr/local/makerspaceleiden-crm
  git pull <right version and branch>
 
Activate the right vertual environment (rebuild with python3 -menv venv):

  source venv/bin/activate

If needed - make a backup of the database with either

  python3 manage.py dumpdata

or (passowrd in makerspaceleiden/my.cnf):

  mysqldump -u mslcrmuser  -p mslcrm

Check for any new static stuff & updates:

  pip3 install -r requirements.txt 

Next migrate the database if needed:
  
  python3 manage.py makemigrations
  python3 manage.py migrate

If the asset changes - then also run the collect static:

  python3 manage.py collectstatic --dry-run --noinput

examine the output and then either rerun the command or manually adjust.

And finally restart the webserver:

   sudo apachectl restart

'''NOTE: var/media contains the images.'''

So while you can actually wack & redo the entire setup - you will need to preserve var/media. (Perhaps we should move var/media to /var/media - fully outside the tree - as it is also the only thing that 'www-data' can write to.

== In production - setup from zero ==

Note: '''pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).'''

Make sure the baseline tools are present:

   sudo apt-get install python3 libmysqlclient  default-libmysqlclient-dev msmtp-mta apache2 libapache2-mod-uwsgi

The  default-libmysqlclient-dev module is needed as pip3 wants mysqlconfig - which is not in the baseline libmysqlclient.

Initial checkout of code:

    cd /usr/local
    git clone https://github.com/dirkx/makerspaceleiden-crm.git

Make evertyin group owned (crmadmin) and add that group to the accounts
of those that need to maintain it.

Create random seed
    openssl rand 128 > /etc/crm_secret_key.txt
    chmod 640 /etc/crm_secret_key.txt
    chgrp www-data /etc/crm_secret_key.txt
    
Allow storing of uploads in media by the suid  that the webserver runs as:

    mkdir -p var/media
    chown www-data var/media
    
Allow server to rotate/recreate logfiles on the fly

    mkdir /var/log/crm
    chown www-data:crmadmin /var/log/crm
    chmod 770 /var/log/crm

Prepare env for python and pull in the various dependencies.

    python3 -mvenv venv
    source ./venv/bin/activate
    pip3 install -r requirements
    
Set up the framework

    cd makerspaceleiden
    ln -s prod.py local.py
    cat > makerspaceledien/my.cnf <<EOM
    [client]
    database = mslcrm
    user = mslcrmuser
    password = XXXX-passowrd-XXX
    default-character-set = utf8
    EOM
    
Create database & user

    mysql (suply database admin arguments as and when needed)
    create database mslcrm;
    create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
    grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
    flush priveleges;
    
Check for issues, init and build database & site

    python3 manage.py check --deploy
    python3 manage.py makemigrations
    python3 manage.py migrate
    python3 manage.py collectstatic
    
Check that email works:

   python3 manage.py sendtestemail your@email.address.com

Create temp super user so you can log into complete the setup

    python3 manage.py createsuperuser
    
Create apache config

    cat > /etc/apache2/sites-available/crm.conf <<EOM
    WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
    
    WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
    WSGIProcessGroup  crm 
    
    Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
    Alias /media/      /usr/local/makerspaceleiden-crm/var/media/
    
    <Directory /usr/local/makerspaceleiden-crm/>
      <Files wsgi.py>
         Require all granted
      </Files>
    </Directory>
    
    <Directory /usr/local/makerspaceleiden-crm/static>
            Options None
            order deny,allow
            allow from all
            Require all granted
    </Directory>
    
    <Directory /usr/local/makerspaceleiden-crm/var/media>
            Options None
            order deny,allow
            allow from all
            Require all granted
    </Directory>
    EOM

Activate this setup

    ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf 
    
Start server and keep an eye on the log:

    apacheclt configtest
    apachectl restart
    tail -F /var/log/apache2/error.log

Now go to https://makerspaceleiden.nl/crm and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.

== Setup Signal Bridge ==

Requires openjdk, java-dbus-bin and dbussy. Binary package from Shttps://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).

Post install - as user signal-cli - activation is required:

   $ signal-cli -u +317... register -voice
   $ signal-cli -u +317... verify code from call

which will create the right structure in ~/.local. 

Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.

== Logfile retention ==

Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.

See https://github.com/dirkx/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.

== Cleanup and reminder crons ==

To be configured/written

== Backup ==

This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database
and captures the whole directory -- which includes var/media -- the uploads.

See the section [[MTA-Setup#Backup | backup]] on the general [[MTA-Setup| Server setup page.]]

== Mailing list integration ==

Mailing lists are ran by a third party - we currently use a hack to manage these by faking browser access to their form API.

=== experimental update ===

Integration via their newly fangled "Sympa" web service.

=== Security/Privacy consideration ===

Our mailing list provider is a processed of our data; and distributes the email to our participants. This is the basis for allowing this party access to the email addresses or our participants - i.e. we only share for that purpose. 

With this new interface - we can minimise the information to exactly that. And we stop sharing non-subscribers (which previously had to have nomail set).

Additional measures:

# Gather the /just/ needed data in a single purpose view (minimise)
# Reduce the need for a copy at the provider (no retention at processor)
# Specific user that is restricted to just that view and has only select/view; no write.
# Strong password & secure transfer of the password to the third party (gpg)
# Require the use of SSL/TLS for this user in mysql (protect data and password)
# Lock down of this user to a specific IP in mysql
# Use of IP tables to lock down the connection as a second backstop
# Move to a non-common port (fluff)
# Data processing agreement part of contract with provider. Provider selected as it has an EU presence and sees this core to their commercial offering.

Still open:

# Does the API still need the (full) name - as mailman currently does - for use in the From: ? Or can we further limit this.
# Can we use X.509 client certs for authentication.
# how do we do digest ?

=== configuration ===
Create a mysql user specific to sympa; $SYMPA_IP is the IP address of their gateway (currently just one) and $secret is a strong password:

     create user mlist@$SYMPA_IP identified by '$secret' REQUIRE SSL;

Then create a view with just the data needed:

     create view mailinglists_api as
        select distinct 
                   concat(first_name,' ',last_name) "comment_subscriber",
                   name "list", 
                   email "user_subscriber",
                   CASE
                     WHEN mailinglists_subscription.digest
                     THEN "digestplain"
                     ELSE "mail"
                   END as "digest"
        from    members_user,mailinglists_subscription,mailinglists_mailinglist
        where   mailinglists_subscription.member_id = members_user.id and
                mailinglists_subscription.mailinglist_id = mailinglists_mailinglist.id and
                mailinglists_subscription.active = 1
        order by email
        with read only;

Add to above

       and mailinglists_subscription.name = 'testlist'

to curtail it to just a test list of so required. And grant this integration user access to just that view:

     GRANT SELECT, SHOW VIEW ON mslcrm.mailinglists_api TO mlist@ $SYMPA_IP;

This then in effect gives the following to the right user:

     mysql> select distinct * from mailinglists_api where list='testlist';
     +-----------------------+------------+----------------------+
     | comment_subscriber.   | list       | user_subscriber      | ....
     +-----------------------+------------+----------------------+
     | Ted de Tester         | testlist   | tester@xxxx.xxxx     |....
     ...... etc.......................................................
     +-----------------------+------------+----------------------+
     112 row in set (0.04 sec)

Note that you can use:

      SELECT host,user FROM mysql.user where user = 'mlist';
      SELECT CONCAT('SHOW GRANTS FOR ' ' ',user,' ' '@' ' ',host,' ' ';') FROM mysql.user;

to check permissions and accounts. Then provide the mailinglist provider with the following file/data:

    db_type mysql
    db_port $PORT 
    db_host $OUR_IP
    db_user mlist
    db_passwd $secret
    db_name mslcrm
    sql_query select distinct email from mailinglists_api where list = 'spacelog';

Their gpg/pgp key is at https://www.mailmanlists.net/pgppubkey.mailmanlists.support.

Configure SSL on the right port in /etc/mysql/mysql.conf.d/mysqld.conf

    [mysqld]
    ...
    bind-address            = 127.0.0.1, $OUR_IP
    port=$PORT
    .....
    ssl_ca=msl-ca.pem
    ssl_cert=msl-cert.pem
    ssl_key=msl-cert.key
    require_secure_transport=ON

The above keys need to be mysql:mysql and 600 (a shared group does not seem to work, nor does a path; perhaps some chroot() going on) so this is set up by adding the script below to /etc/letsencrypt/renewal-hooks/deploy/:

     #!/bin/sh
     set -e
     cd /var/lib/mysql
     
     cp /etc/letsencrypt/live/makerspaceleiden.nl/cert.pem     msl-cert.pem
     cp /etc/letsencrypt/live/makerspaceleiden.nl/privkey.pem  msl-cert.key
     cp /etc/letsencrypt/live/makerspaceleiden.nl/chain.pem    msl-ca.pem
      
     chown mysql:mysql msl-*
     chmod 600 msl-*
     
     exit 0

And allow acccess:

     sudo ufw allow in from $SYMPA_IP to any port $PORT

or

    iptables -A INPUT  -p tcp -s $SYMPA_IP --sport 1024:65535 -d $OUR_IP --dport $PORT \
           -m state --state NEW,ESTABLISHED -j ACCEPT
    iptables -A OUTPUT -p tcp -s 2$OUR_IP --sport $PORT -d $SYMPA_IP --dport 1024:65535 \
           -m state --state ESTABLISHED -j ACCEPT