Website mijn.makerspaceleiden.nl setup: verschil tussen versies

Uit MakerSpace Leiden
Naar navigatie springen Naar zoeken springen
Geen bewerkingssamenvatting
Luke (overleg | bijdragen)
adds systemd documentation, resolving lingering todo
 
(83 tussenliggende versies door 7 gebruikers niet weergegeven)
Regel 1: Regel 1:
== On a demo/local laptop ==
[[Category:Servers network & websites]]


    git clone https://github.com/dirkx/makerspaceleiden-crm.git
    cd makerspaceleiden-crm
    sh loaddemo.sh


then visit http://localhost:8000/
The code is hosted on GitHub at:
https://github.com/MakerSpaceLeiden/makerspaceleiden-crm


Prepare for restarts with
[https://github.com/MakerSpaceLeiden/makerspaceleiden-crm?tab=readme-ov-file#installation-options Instructions] for getting up and running locally are available in the Read Me.
  source venv/bin/activate


and then do :
== Initial Setup ==
  python3 manage.py runsever


as needed
Note: '''pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).'''


== Om production -- update ==
=== Prerequisites ===


  cd /usr/local/makerspaceleiden-crm
Make sure the baseline tools are present:
  git pull <right version>
 
  source venv/bin/activate
  python3 manage.py makemigrations
  python3 manage.py migrate


== In production - setup from zero
<syntaxhighlight lang="bash">
''''
sudo apt-get install msmtp-mta apache2 libapache2-mod-uwsgi
sudo snap install astral-uv --classic
</syntaxhighlight>
 
=== Code Setup ===
 
Initial checkout of code:
 
<syntaxhighlight lang="bash">
cd /usr/local
cd /usr/local
git clone https://github.com/dirkx/makerspaceleiden-crm.git
git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git
</syntaxhighlight>


# Create random seed
Make everything group owned (crmadmin) and add that group to the accounts
of those that need to maintain it.
 
=== Security Configuration ===
 
Create random seed:
<syntaxhighlight lang="bash">
openssl rand 128 > /etc/crm_secret_key.txt
openssl rand 128 > /etc/crm_secret_key.txt
chmod 640 /etc/crm_secret_key.txt
chmod 640 /etc/crm_secret_key.txt
chgrp www-data /etc/crm_secret_key.txt
chgrp www-data /etc/crm_secret_key.txt
</syntaxhighlight>
   
Allow storing of uploads in media by the suid that the webserver runs as:


# Allow storing of uploads in media by the suid
<syntaxhighlight lang="bash">
# all this runs as.
mkdir -p var/media
mkdir -p var/media
chown www-data var/media
chown www-data var/media
</syntaxhighlight>
   
Allow server to rotate/recreate logfiles on the fly:


# Prepare env for python and pull in the various dependencies.
<syntaxhighlight lang="bash">
python3 -mvenv venv
mkdir /var/log/crm
source ./venv/bin/activate
chown www-data:crmadmin /var/log/crm
pip3 install -r requirements
chmod 770 /var/log/crm
</syntaxhighlight>


# Set up the framework
=== Python Environment ===
 
Prepare environment for python and pull in the various dependencies:
 
<syntaxhighlight lang="bash">
uv venv
source ./.venv/bin/activate
uv sync
</syntaxhighlight>
   
Set up the framework:
 
<syntaxhighlight lang="bash">
cd makerspaceleiden
cd makerspaceleiden
ln -s prod.py local.py
ln -s prod.py local.py
Regel 54: Regel 78:
default-character-set = utf8
default-character-set = utf8
EOM
EOM
</syntaxhighlight>
   
=== Database Setup ===
Create database & user:
<syntaxhighlight lang="sql">
-- mysql (suply database admin arguments as and when needed)
CREATE DATABASE mslcrm;
CREATE USER 'mslcrmuser'@'localhost' IDENTIFIED BY 'XXXX-passowrd-XXX';
GRANT ALL PRIVILEGES ON mslcrm.* TO 'mslcrmuser'@'localhost';
FLUSH PRIVILEGES;
</syntaxhighlight>
   
Check for issues, init and build database & site:
<syntaxhighlight lang="bash">
uv run manage.py check --deploy
uv run manage.py makemigrations
uv run manage.py migrate
uv run manage.py collectstatic
</syntaxhighlight>
   
Check that email works:
<syntaxhighlight lang="bash">
uv run manage.py sendtestemail your@email.address.com
</syntaxhighlight>
Create temp super user so you can log into complete the setup:


# Create database & user
<syntaxhighlight lang="bash">
mysql (suply database admin arguments as and when needed)
uv run manage.py createsuperuser
create database mslcrm;
</syntaxhighlight>
create user 'mslcrmuser'@'localhost' identified by 'XXXX-passowrd-XXX';
grant all priveleges on mslcrm.* to 'mslcrmuser'@'localhost';
flush priveleges;


# Check for issues and build database & site
=== Web Server Configuration ===
python manage.py check --deploy
python3 manage.py makemigrations
python3 manage.py migrate
python manage.py collectstatic


# Create temp super user
Create apache config:
python3 manage.py createsuperuser


# Create apache config
<syntaxhighlight lang="apache">
cat > /etc/apache2/sites-available/crm.conf <<EOM
cat > /etc/apache2/sites-available/crm.conf <<EOM
WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm
Regel 101: Regel 147:
</Directory>
</Directory>
EOM
EOM
</syntaxhighlight>
Activate this setup:
<syntaxhighlight lang="bash">
ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf  
ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf  
</syntaxhighlight>
   
Start server and keep an eye on the log:


# Start server
<syntaxhighlight lang="bash">
#
apachectl configtest
apacheclt configtest
apachectl restart
apachectl restart
tail -F /var/log/apache2/error.log
tail -F /var/log/apache2/error.log
</syntaxhighlight>
=== Daphne Setup ===
Daphne is the ASGI server for the Django application. It is used to serve the application over HTTP. The following is a systemd service file `/etc/systemd/system/crm-daphne.service`:
<syntaxhighlight lang="bash">
[Unit]
Description=Daphne Webserver for CRM
After=network.target
[Service]
Environment="AGGREGATOR_BASE_URL=http://127.0.0.1:**port**"
Environment="AGGREGATOR_USERNAME=**username**"
Environment="AGGREGATOR_PASSWORD==**password**"
Environment="FORCE_SCRIPT_NAME="
Environment="LOGIN_URL=/login"
Environment="LOGIN_REDIRECT_URL=/"
Environment="LOGOUT_REDIRECT_URL=/"
Environment="LOG_FILE_NAME=/var/log/crm/daphne-crm-debug.log"
Environment="UV_CACHE_DIR=/var/cache/crm/uv"
Environment="UV_PYTHON_INSTALL_DIR=/var/cache/crm/uv"
Environment="STATIC_URL=/static"
Type=simple
User=www-data
Group=crmadmin
WorkingDirectory=/usr/local/makerspaceleiden-crm
ExecStart=/usr/local/bin/uv run daphne --access-log /var/log/crm//daphne-access.log -b 127.0.0.1 -p 8000 -v 0 makerspaceleiden.asgi:application
[Install]
WantedBy=multi-user.target
</syntaxhighlight>
Enable and start the service:
<syntaxhighlight lang="bash">
sudo systemctl enable crm-daphne
sudo systemctl start crm-daphne
</syntaxhighlight>
Check the status:
<syntaxhighlight lang="bash">
sudo systemctl status crm-daphne
</syntaxhighlight>
=== Initial Configuration ===
Now go to https://mijn.makerspaceleiden.nl/ and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.
== Configuration & Integration ==
=== Signal Bridge Setup ===
Requires openjdk, java-dbus-bin and dbussy. Binary package from https://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).
Post install - as user signal-cli - activation is required:
<syntaxhighlight lang="bash">
signal-cli -u +317... register -voice
signal-cli -u +317... verify code from call
</syntaxhighlight>
which will create the right structure in ~/.local.
Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.
=== Configurating External Services ===
Mailing lists are ran by a third party - sympalists.net. The integration via their newly fangled "Sympa" web service -- See [[Sympa mailing list setup]] for details.
Query to find duplicates in the subscriptions:
<syntaxhighlight lang="sql">
SELECT *
FROM mailinglists_subscription
WHERE id NOT IN (
    SELECT MAX(id)
    FROM mailinglists_subscription
    GROUP BY mailinglist_id, member_id
);
</syntaxhighlight>
== Maintenance & Monitoring ==
=== Deploying Code Changes ===
Check that you are in the 'crmadmin' group.
<syntaxhighlight lang="bash">
cd /usr/local/makerspaceleiden-crm
# Ensure you are on the correct branch
git checkout prod
# Pull the latest changes
git pull
# Run the deployment script
./rollout-prod.sh
</syntaxhighlight>
=== Logfile Retention ===
Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.
See https://github.com/MakerSpaceLeiden/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.
=== Automated Maintenance ===
The following entries need to be added to the crontab (either to /etc/cron, to cron.daily/monthly or to the `users' cron):
<syntaxhighlight lang="bash">
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=noc@makerspaceleiden.nl
MAILFROM="MSL Server (cron)"
NULLMAILUSER_NAME="MSL Server (cron)"
....
# General maintenance mails of [https://mijn.makerspaceleiden.nl mijn.makerspaceleiden.nl].
#
# m h d m d user  cmd
1 1 * * * USER test -x /usr/local/makerspaceleiden-crm/cron-midnight.sh && /usr/local/makerspaceleiden-crm/cron-midnight.sh
1 2 1 * * USER test -x /usr/local/makerspaceleiden-crm/cron-monthly.sh && /usr/local/makerspaceleiden-crm/cron-monthly.sh
</syntaxhighlight>
With 'USER' a user with the right rights.
=== Backup ===
This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database
and captures the whole directory -- which includes var/media -- the uploads.


# Create the initial members/structure. Assign the trusteeds
See the section [[Server setup#Backup | backup]] on the general [[Server setup| Server setup page.]]
# the super user permission; then delete the temporary admin you made.

Huidige versie van 15 jun 2025 20:30


The code is hosted on GitHub at: https://github.com/MakerSpaceLeiden/makerspaceleiden-crm

Instructions for getting up and running locally are available in the Read Me.

Initial Setup

Note: pretty much everything below is stock/totally-standard django/python Standard Operating Procedure (except for the chmod/chgrp on the var/media upload).

Prerequisites

Make sure the baseline tools are present:

sudo apt-get install msmtp-mta apache2 libapache2-mod-uwsgi
sudo snap install astral-uv --classic

Code Setup

Initial checkout of code:

cd /usr/local
git clone https://github.com/MakerSpaceLeiden/makerspaceleiden-crm.git

Make everything group owned (crmadmin) and add that group to the accounts of those that need to maintain it.

Security Configuration

Create random seed:

openssl rand 128 > /etc/crm_secret_key.txt
chmod 640 /etc/crm_secret_key.txt
chgrp www-data /etc/crm_secret_key.txt

Allow storing of uploads in media by the suid that the webserver runs as:

mkdir -p var/media
chown www-data var/media

Allow server to rotate/recreate logfiles on the fly:

mkdir /var/log/crm
chown www-data:crmadmin /var/log/crm
chmod 770 /var/log/crm

Python Environment

Prepare environment for python and pull in the various dependencies:

uv venv
source ./.venv/bin/activate
uv sync

Set up the framework:

cd makerspaceleiden
ln -s prod.py local.py
cat > makerspaceledien/my.cnf <<EOM
[client]
database = mslcrm
user = mslcrmuser
password = XXXX-passowrd-XXX
default-character-set = utf8
EOM

Database Setup

Create database & user:

-- mysql (suply database admin arguments as and when needed)
CREATE DATABASE mslcrm;
CREATE USER 'mslcrmuser'@'localhost' IDENTIFIED BY 'XXXX-passowrd-XXX';
GRANT ALL PRIVILEGES ON mslcrm.* TO 'mslcrmuser'@'localhost';
FLUSH PRIVILEGES;

Check for issues, init and build database & site:

uv run manage.py check --deploy
uv run manage.py makemigrations
uv run manage.py migrate
uv run manage.py collectstatic

Check that email works:

uv run manage.py sendtestemail your@email.address.com

Create temp super user so you can log into complete the setup:

uv run manage.py createsuperuser

Web Server Configuration

Create apache config:

cat > /etc/apache2/sites-available/crm.conf <<EOM
WSGIScriptAlias /crm /usr/local/makerspaceleiden-crm/makerspaceleiden/wsgi.py  process-group=crm

WSGIDaemonProcess crm python-home=/usr/local/makerspaceleiden-crm/venv python-path=/usr/local/makerspaceleiden-crm/
WSGIProcessGroup  crm 

Alias /crm-static/ /usr/local/makerspaceleiden-crm/static/
Alias /media/      /usr/local/makerspaceleiden-crm/var/media/

<Directory /usr/local/makerspaceleiden-crm/>
  <Files wsgi.py>
     Require all granted
  </Files>
</Directory>

<Directory /usr/local/makerspaceleiden-crm/static>
        Options None
        order deny,allow
        allow from all
        Require all granted
</Directory>

<Directory /usr/local/makerspaceleiden-crm/var/media>
        Options None
        order deny,allow
        allow from all
        Require all granted
</Directory>
EOM

Activate this setup:

ln -s /etc/apache2/sites-available/crm.conf  /etc/apache2/sites-active/crm.conf

Start server and keep an eye on the log:

apachectl configtest
apachectl restart
tail -F /var/log/apache2/error.log

Daphne Setup

Daphne is the ASGI server for the Django application. It is used to serve the application over HTTP. The following is a systemd service file `/etc/systemd/system/crm-daphne.service`:

[Unit]
Description=Daphne Webserver for CRM
After=network.target

[Service]
Environment="AGGREGATOR_BASE_URL=http://127.0.0.1:**port**"
Environment="AGGREGATOR_USERNAME=**username**"
Environment="AGGREGATOR_PASSWORD==**password**"
Environment="FORCE_SCRIPT_NAME="
Environment="LOGIN_URL=/login"
Environment="LOGIN_REDIRECT_URL=/"
Environment="LOGOUT_REDIRECT_URL=/"
Environment="LOG_FILE_NAME=/var/log/crm/daphne-crm-debug.log"
Environment="UV_CACHE_DIR=/var/cache/crm/uv"
Environment="UV_PYTHON_INSTALL_DIR=/var/cache/crm/uv"
Environment="STATIC_URL=/static"
Type=simple
User=www-data
Group=crmadmin
WorkingDirectory=/usr/local/makerspaceleiden-crm
ExecStart=/usr/local/bin/uv run daphne --access-log /var/log/crm//daphne-access.log -b 127.0.0.1 -p 8000 -v 0 makerspaceleiden.asgi:application

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl enable crm-daphne
sudo systemctl start crm-daphne

Check the status:

sudo systemctl status crm-daphne

Initial Configuration

Now go to https://mijn.makerspaceleiden.nl/ and create the initial members/structure. Assign the trustees the super user permission; then delete the temporary admin you made.

Configuration & Integration

Signal Bridge Setup

Requires openjdk, java-dbus-bin and dbussy. Binary package from https://github.com/AsamK/signal-cli/; installed in /home/signal-cli (post install 600/700 locked down).

Post install - as user signal-cli - activation is required:

signal-cli -u +317... register -voice
signal-cli -u +317... verify code from call

which will create the right structure in ~/.local.

Source is required for the systemd files - installed with https://github.com/AsamK/signal-cli/wiki/DBus-service instructions.

Configurating External Services

Mailing lists are ran by a third party - sympalists.net. The integration via their newly fangled "Sympa" web service -- See Sympa mailing list setup for details.

Query to find duplicates in the subscriptions:

SELECT * 
FROM mailinglists_subscription 
WHERE id NOT IN (
    SELECT MAX(id) 
    FROM mailinglists_subscription 
    GROUP BY mailinglist_id, member_id
);

Maintenance & Monitoring

Deploying Code Changes

Check that you are in the 'crmadmin' group.

cd /usr/local/makerspaceleiden-crm
# Ensure you are on the correct branch
git checkout prod
# Pull the latest changes
git pull
# Run the deployment script
./rollout-prod.sh

Logfile Retention

Configured in 'prod' settings - few MBs/days of logs is kept & then rotated out/deleted using the standard RotatingLogging handler.

See https://github.com/MakerSpaceLeiden/makerspaceleiden-crm/commit/de5de35dd22a317bb93568dadc1b8737a819023a for details.

Automated Maintenance

The following entries need to be added to the crontab (either to /etc/cron, to cron.daily/monthly or to the `users' cron):

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=noc@makerspaceleiden.nl
MAILFROM="MSL Server (cron)"
NULLMAILUSER_NAME="MSL Server (cron)"
....
# General maintenance mails of [https://mijn.makerspaceleiden.nl mijn.makerspaceleiden.nl].
#
# m h d m d user  cmd
1 1 * * * USER test -x /usr/local/makerspaceleiden-crm/cron-midnight.sh && /usr/local/makerspaceleiden-crm/cron-midnight.sh
1 2 1 * * USER test -x /usr/local/makerspaceleiden-crm/cron-monthly.sh && /usr/local/makerspaceleiden-crm/cron-monthly.sh

With 'USER' a user with the right rights.

Backup

This is done by the /etc/duplicity/run.sh setup (along with everything else). It does a dump of the MySQL database and captures the whole directory -- which includes var/media -- the uploads.

See the section backup on the general Server setup page.