Website mijn.makerspaceleiden.nl setup: verschil tussen versies

Uit MakerSpace Leiden
Naar navigatie springen Naar zoeken springen
Geen bewerkingssamenvatting
Luke (overleg | bijdragen)
Geen bewerkingssamenvatting
 
(77 tussenliggende versies door 6 gebruikers niet weergegeven)
Regel 1: Regel 1:
== On a demo/local laptop ==
[[Category:Servers network & websites]]


    git clone https://github.com/dirkx/makerspaceleiden-crm.git
The code is hosted on GitHub at:
    cd makerspaceleiden-crm
https://github.com/MakerSpaceLeiden/makerspaceleiden-crm
    sh loaddemo.sh


then visit http://localhost:8000/
[https://github.com/MakerSpaceLeiden/makerspaceleiden-crm?tab=readme-ov-file#installation-options Instructions] for getting up and running locally are available in the Read Me.
 
 
== On the production server -- how to update ==
 
Check that you are in the 'crmadmin' group. Then:
 
  cd /usr/local/makerspaceleiden-crm
  git pull <right version and branch>
Activate the right vertual environment (rebuild with python3 -menv venv):


Prepare for restarts with
   source venv/bin/activate
   source venv/bin/activate


and then do :
If needed - make a backup of the database with either
  python3 manage.py runsever


as needed
  python3 manage.py dumpdata


== Om production -- update ==
or (passowrd in makerspaceleiden/my.cnf):


   cd /usr/local/makerspaceleiden-crm
   mysqldump -u mslcrmuser  -p mslcrm
   git pull <right version>
 
Update PIP if you get warnings first:
 
   python3.8 -m pip install --upgrade pip
 
Check for any new static stuff & updates:
 
  pip3 list --outdated
  pip3 install -r requirements.txt  --upgrade
 
Next migrate the database if needed:
    
    
  source venv/bin/activate
   python3 manage.py makemigrations
   python3 manage.py makemigrations
   python3 manage.py migrate
   python3 manage.py migrate


== In production - setup from zero
If the asset changes - then also run the collect static:
''''
 
cd /usr/local
  python3 manage.py collectstatic --dry-run --noinput
git clone https://github.com/dirkx/makerspaceleiden-crm.git
 
examine the output and then either rerun the command or manually adjust.
 
And finally restart the webserver:
 
  sudo apachectl restart
 
'''NOTE: var/media contains the images.'''
 
'''XXX todo XXXXX - daphne needs to be setup/restarted and log files - including systemd setup !
'''
 
So while you can actually wack & redo the entire setup - you will need to preserve var/media. (Perhaps we should move var/media to /var/media - fully outside the tree - as it is also the only thing that 'www-data' can write to.
 
== In production - setup from zero ==
 
Note: '''pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).'''
 
Make sure the baseline tools are present:
 
  sudo apt-get install python3 libmysqlclient  default-libmysqlclient-dev msmtp-mta apache2 libapache2-mod-uwsgi
 
The  default-libmysqlclient-dev module is needed as pip3 wants mysqlconfig - which is not in the baseline libmysqlclient.
 
Initial checkout of code:
 
    cd /usr/local
    git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git
 
Make evertyin group owned (crmadmin) and add that group to the accounts
of those that need to maintain it.
 
Create random seed
    openssl rand 128 > /etc/crm_secret_key.txt
    chmod 640 /etc/crm_secret_key.txt
    chgrp www-data /etc/crm_secret_key.txt
   
Allow storing of uploads in media by the suid  that the webserver runs as:
 
    mkdir -p var/media
    chown www-data var/media
   
Allow server to rotate/recreate logfiles on the fly
 
    mkdir /var/log/crm
    chown www-data:crmadmin /var/log/crm
    chmod 770 /var/log/crm
 
Prepare env for python and pull in the various dependencies.
 
    python3 -mvenv venv
    source ./venv/bin/activate
    pip3 install -r requirements
   
Set up the framework
 
    cd makerspaceleiden
    ln -s prod.py local.py
    cat > makerspaceledien/my.cnf <<EOM
    [client]
    database = mslcrm
    user = mslcrmuser
    password = XXXX-passowrd-XXX
    default-character-set = utf8
    EOM
   
Create database & user
 
    mysql (suply database admin arguments as and when needed)
    create database mslcrm;
    create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
    grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
    flush priveleges;
   
Check for issues, init and build database & site
 
    python3 manage.py check --deploy
    python3 manage.py makemigrations
    python3 manage.py migrate
    python3 manage.py collectstatic
   
Check that email works:
 
  python3 manage.py sendtestemail your@email.address.com
 
Create temp super user so you can log into complete the setup
 
    python3 manage.py createsuperuser
   
Create apache config
 
    cat > /etc/apache2/sites-available/crm.conf <<EOM
    WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
   
    WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
    WSGIProcessGroup  crm
   
    Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
    Alias /media/      /usr/local/makerspaceleiden-crm/var/media/
   
    <Directory /usr/local/makerspaceleiden-crm/>
      <Files wsgi.py>
        Require all granted
      </Files>
    </Directory>
   
    <Directory /usr/local/makerspaceleiden-crm/static>
            Options None
            order deny,allow
            allow from all
            Require all granted
    </Directory>
   
    <Directory /usr/local/makerspaceleiden-crm/var/media>
            Options None
            order deny,allow
            allow from all
            Require all granted
    </Directory>
    EOM
 
Activate this setup
 
    ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf
   
Start server and keep an eye on the log:
 
    apacheclt configtest
    apachectl restart
    tail -F /var/log/apache2/error.log
 
'''XXX todo XXXXX - daphne needs to be setup/restarted and log files - including systemd setup !
'''
 
 
Now go to https://mijn.makerspaceleiden.nl/ and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.
 
== Setup Signal Bridge ==
 
Requires openjdk, java-dbus-bin and dbussy. Binary package from Shttps://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).
 
Post install - as user signal-cli - activation is required:
 
  $ signal-cli -u +317... register -voice
  $ signal-cli -u +317... verify code from call
 
which will create the right structure in ~/.local.


# Create random seed
Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.
openssl rand 128 > /etc/crm_secret_key.txt
chmod 640 /etc/crm_secret_key.txt
chgrp www-data /etc/crm_secret_key.txt


# Allow storing of uploads in media by the suid
== Logfile retention ==
# all this runs as.
mkdir -p var/media
chown www-data var/media


# Prepare env for python and pull in the various dependencies.
Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.
python3 -mvenv venv
source ./venv/bin/activate
pip3 install -r requirements


# Set up the framework
See https://github.com/MakerSpaceLeiden/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.
cd makerspaceleiden
ln -s prod.py local.py
cat > makerspaceledien/my.cnf <<EOM
[client]
database = mslcrm
user = mslcrmuser
password = XXXX-passowrd-XXX
default-character-set = utf8
EOM


# Create database & user
== Cleanup and reminder crons ==
mysql (suply database admin arguments as and when needed)
create database mslcrm;
create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
flush priveleges;


# Check for issues and build database & site
The following entries need to be added to the crontab (either to /etc/cron, to cron.daily/monthly or to the `users' cron):
python manage.py check --deploy
python3 manage.py makemigrations
python3 manage.py migrate
python manage.py collectstatic


# Create temp super user
    SHELL=/bin/sh
python3 manage.py createsuperuser
    PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
    MAILTO=noc@makerspaceleiden.nl
    MAILFROM="MSL Server (cron)"
    NULLMAILUSER_NAME="MSL Server (cron)"
    ....
    # General maintenance mails of [https://mijn.makerspaceleiden.nl mijn.makerspaceleiden.nl].
    #
    # m h d m d user cmd
    1 1 * * * USER test -x /usr/local/makerspaceleiden-crm/cron-midnight.sh && /usr/local/makerspaceleiden-crm/cron-midnight.sh
    1 2 1 * * USER test -x /usr/local/makerspaceleiden-crm/cron-monthly.sh && /usr/local/makerspaceleiden-crm/cron-monthly.sh


# Create apache config
With 'USER' a user with the right rights.
cat > /etc/apache2/sites-available/crm.conf <<EOM
WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm


WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
== Backup ==
WSGIProcessGroup  crm


Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database
Alias /media/      /usr/local/makerspaceleiden-crm/var/media/
and captures the whole directory -- which includes var/media -- the uploads.


<Directory /usr/local/makerspaceleiden-crm/>
See the section [[Server setup#Backup | backup]] on the general [[Server setup| Server setup page.]]
  <Files wsgi.py>
    Require all granted
  </Files>
</Directory>


<Directory /usr/local/makerspaceleiden-crm/static>
== Mailing list integration ==
        Options None
        order deny,allow
        allow from all
        Require all granted
</Directory>


<Directory /usr/local/makerspaceleiden-crm/var/media>
Mailing lists are ran by a third party - sympalists.net. The integration via their newly fangled "Sympa" web service -- See [[Sympa mailing list setup]] for details.
        Options None
        order deny,allow
        allow from all
        Require all granted
</Directory>
EOM
ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf


# Start server
Query to find duplicates in the subscriptions:
#
apacheclt configtest
apachectl restart
tail -F /var/log/apache2/error.log


# Create the initial members/structure. Assign the trusteeds
        select * from mailinglists_subscription where id not in (select max(id) from mailinglists_subscription group by mailinglist_id,member_id);
# the super user permission; then delete the temporary admin you made.

Huidige versie van 20 mei 2025 10:14


The code is hosted on GitHub at: https://github.com/MakerSpaceLeiden/makerspaceleiden-crm

Instructions for getting up and running locally are available in the Read Me.


On the production server -- how to update

Check that you are in the 'crmadmin' group. Then:

 cd /usr/local/makerspaceleiden-crm
 git pull <right version and branch>

Activate the right vertual environment (rebuild with python3 -menv venv):

 source venv/bin/activate

If needed - make a backup of the database with either

 python3 manage.py dumpdata

or (passowrd in makerspaceleiden/my.cnf):

 mysqldump -u mslcrmuser  -p mslcrm

Update PIP if you get warnings first:

 python3.8 -m pip install --upgrade pip

Check for any new static stuff & updates:

 pip3 list --outdated
 pip3 install -r requirements.txt  --upgrade

Next migrate the database if needed:

 python3 manage.py makemigrations
 python3 manage.py migrate

If the asset changes - then also run the collect static:

 python3 manage.py collectstatic --dry-run --noinput

examine the output and then either rerun the command or manually adjust.

And finally restart the webserver:

  sudo apachectl restart

NOTE: var/media contains the images.

XXX todo XXXXX - daphne needs to be setup/restarted and log files - including systemd setup !

So while you can actually wack & redo the entire setup - you will need to preserve var/media. (Perhaps we should move var/media to /var/media - fully outside the tree - as it is also the only thing that 'www-data' can write to.

In production - setup from zero

Note: pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).

Make sure the baseline tools are present:

  sudo apt-get install python3 libmysqlclient  default-libmysqlclient-dev msmtp-mta apache2 libapache2-mod-uwsgi

The default-libmysqlclient-dev module is needed as pip3 wants mysqlconfig - which is not in the baseline libmysqlclient.

Initial checkout of code:

   cd /usr/local
   git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git

Make evertyin group owned (crmadmin) and add that group to the accounts of those that need to maintain it.

Create random seed

   openssl rand 128 > /etc/crm_secret_key.txt
   chmod 640 /etc/crm_secret_key.txt
   chgrp www-data /etc/crm_secret_key.txt
   

Allow storing of uploads in media by the suid that the webserver runs as:

   mkdir -p var/media
   chown www-data var/media
   

Allow server to rotate/recreate logfiles on the fly

   mkdir /var/log/crm
   chown www-data:crmadmin /var/log/crm
   chmod 770 /var/log/crm

Prepare env for python and pull in the various dependencies.

   python3 -mvenv venv
   source ./venv/bin/activate
   pip3 install -r requirements
   

Set up the framework

   cd makerspaceleiden
   ln -s prod.py local.py
   cat > makerspaceledien/my.cnf <<EOM
   [client]
   database = mslcrm
   user = mslcrmuser
   password = XXXX-passowrd-XXX
   default-character-set = utf8
   EOM
   

Create database & user

   mysql (suply database admin arguments as and when needed)
   create database mslcrm;
   create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
   grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
   flush priveleges;
   

Check for issues, init and build database & site

   python3 manage.py check --deploy
   python3 manage.py makemigrations
   python3 manage.py migrate
   python3 manage.py collectstatic
   

Check that email works:

  python3 manage.py sendtestemail your@email.address.com

Create temp super user so you can log into complete the setup

   python3 manage.py createsuperuser
   

Create apache config

   cat > /etc/apache2/sites-available/crm.conf <<EOM
   WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
   
   WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
   WSGIProcessGroup  crm 
   
   Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
   Alias /media/      /usr/local/makerspaceleiden-crm/var/media/
   
   <Directory /usr/local/makerspaceleiden-crm/>
     <Files wsgi.py>
        Require all granted
     </Files>
   </Directory>
   
   <Directory /usr/local/makerspaceleiden-crm/static>
           Options None
           order deny,allow
           allow from all
           Require all granted
   </Directory>
   
   <Directory /usr/local/makerspaceleiden-crm/var/media>
           Options None
           order deny,allow
           allow from all
           Require all granted
   </Directory>
   EOM

Activate this setup

   ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf 
   

Start server and keep an eye on the log:

   apacheclt configtest
   apachectl restart
   tail -F /var/log/apache2/error.log

XXX todo XXXXX - daphne needs to be setup/restarted and log files - including systemd setup !


Now go to https://mijn.makerspaceleiden.nl/ and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.

Setup Signal Bridge

Requires openjdk, java-dbus-bin and dbussy. Binary package from Shttps://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).

Post install - as user signal-cli - activation is required:

  $ signal-cli -u +317... register -voice
  $ signal-cli -u +317... verify code from call

which will create the right structure in ~/.local.

Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.

Logfile retention

Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.

See https://github.com/MakerSpaceLeiden/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.

Cleanup and reminder crons

The following entries need to be added to the crontab (either to /etc/cron, to cron.daily/monthly or to the `users' cron):

    SHELL=/bin/sh
    PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
    MAILTO=noc@makerspaceleiden.nl
    MAILFROM="MSL Server (cron)"
    NULLMAILUSER_NAME="MSL Server (cron)"
    ....
    # General maintenance mails of mijn.makerspaceleiden.nl.
    #
    # m h d m d user  cmd
    1 1 * * * USER test -x /usr/local/makerspaceleiden-crm/cron-midnight.sh && /usr/local/makerspaceleiden-crm/cron-midnight.sh
    1 2 1 * * USER test -x /usr/local/makerspaceleiden-crm/cron-monthly.sh && /usr/local/makerspaceleiden-crm/cron-monthly.sh

With 'USER' a user with the right rights.

Backup

This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database and captures the whole directory -- which includes var/media -- the uploads.

See the section backup on the general Server setup page.

Mailing list integration

Mailing lists are ran by a third party - sympalists.net. The integration via their newly fangled "Sympa" web service -- See Sympa mailing list setup for details.

Query to find duplicates in the subscriptions:

       select * from mailinglists_subscription where id not in (select max(id) from mailinglists_subscription group by mailinglist_id,member_id);