How To Deploy Matrix Server using Ansible and Docker

In our previous guide, we walked through how to run Synapse Matrix homeserver in Docker Containers. Today we will make the task even simpler by deploying the Matrix Server using Ansible and Docker. But before diving into this matter’s crux, we need to define and get familiar with several concepts.

What is Matrix?

Matrix can be defined as a standard protocol used in real-time message relays such as VoIP/WebRTC signalling, Internet of Things communication, messaging and video calls. It works to eradicate the complexity involved with fragmented IP communications. It acts as a generic HTTP messaging and data synchronisation system for the whole web to allow people, services and devices all over the planet to communicate easily. Matrix users are able to communicate with each other without having to care what app the other user is on.

Matrix offers the following features and benefits to users:

• Open source reference implementations of:
  • Clients (Web (React), iOS, Android)
  • Client SDKs (Javascript, Web (React), iOS, Android)
  • Home servers (Synapse)
  • Application Services (bridges to IRC, Slack, Skype, Lync e.t.c)
• The whole ecosystem and community of everyone running Matrix servers and services
• 3rd party contributions of clients, SDKs, servers and services.
• Open Standard HTTP APIs for transferring JSON messages including:
  • Client-Server API: defines how clients communicate with Matrix home servers.
  • Server-Server API – defines the communication between Matrix home servers(how they exchange messages and synchronise history with each other)
  • Application Service API – defines how to extend the functionality of Matrix with ‘integrations’ and bridge to other networks.
  • Modules – specifies features that must be implemented by particular classes of clients.
• The whole ecosystem and community of everyone running Matrix servers and services

What is Synapse?

Setting up a private home server is a good practice among friends, family or colleagues. It can act as the day-to-day medium of communication within your home server while keeping all the data will be safe and secure in your server. A homeserver can be defined as any implementation deployed on a server that can be accessed through any Matrix client such as Element.

Synapse is one of the popular Matrix home servers developed in 2014 by Matrix.org Foundation. This tool is intended to showcase the concept of Matrix and allow users to run their own home servers which will generally help bootstrap the ecosystem.

Now let’s dive in!

Setup Pre-requisites

The Matrix Server Ansible playbooks best work on an x86 server(recommended) with any of the below operating systems:

• CentOS (7 only for now; 8 is not yet supported)
• Debian (10/Buster or newer)
• Ubuntu (18.04 or newer)
• Archlinux

This guide requires the following installed and configured:

• Python installed on the server for Ansible
• Ansible to run the playbooks
• git
• make
• DNS records for your Domain

These packages can be installed in your workspace as shown:

Install Python git and make:

##On Debian/Ubuntu
sudo apt update
sudo apt install python3 python3-pip git make -y

##On RHEL/Rocky Linux/Alma Linux
sudo yum install python3 python3-pip git make -y

If you have multiple versions of Python3, set the default version as shown:

$ sudo update-alternatives --config python3
There are 2 programs which provide 'python3'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/bin/python3.8
   2           /usr/bin/python3.9

Enter to keep the current selection[+], or type selection number: 2

Verify the made changes:

$ python3 --version
Python 3.9.14

You can use the installed Python-PIP to easily install ansible on your system. This can be done using the command:

sudo pip3 install ansible

You can also install ansible from package managers:

##On Ubuntu / Debian
sudo apt install ansible

##On RHEL/Rocky Linux/Alma Linux
sudo yum install epel-release
sudo yum install ansible ansible-core

Once the installation is complete, verify the version:

$ ansible --version
ansible [core 2.14.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alma9/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/alma9/.ansible/collections:/usr/share/ansible/collections
  executable location = /bin/ansible
  python version = 3.9.14 (main, Nov  7 2022, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True

1. Configure DNS for Matrix Server

For this deployment, we will use several ports. The playbook will install docker and configure the server’s internal firewall for you. But if you have another firewall enabled, you need to allow these ports through it:

• 80/tcp: HTTP webserver
• 443/tcp: HTTPS webserver
• 3478/tcp: TURN over TCP (used by Coturn)
• 3478/udp: TURN over UDP (used by Coturn)
• 5349/tcp: TURN over TCP (used by Coturn)
• 5349/udp: TURN over UDP (used by Coturn)
• 8448/tcp: Matrix Federation API HTTPS web server
• 49152-49172/udp: TURN over UDP

Now you also need to configure DNS for your server. The following DNS settings are enabled by default:

In case you are using Cloudflare DNS, you need to disable the proxy and set all records to DNS only, otherwise fetching certificates might fail. There are some other DNS settings that are optional.

For this guide we will set up DNS by editing the /etc/host file as shown:

$ sudo vim /etc/hosts
192.168.205.2 matrix.geeksforgeeks.org element.geeksforgeeks.org

Replace 192.168.205.2 with the IP address of the server you intend to run the deployment on(managed node) in the above file. You also need to provide your desired domain name in place of geeksforgeeks.org. This ensures that your host will resolve requests for both matrix and element subdomains.

2. Configure Ansible for Matrix Server

We will begin by downloading the playbooks and all the configurations by cloning the repository below on our workspace:

git clone https://github.com/spantaleev/matrix-docker-ansible-deploy.git

Once downloaded, navigate into the directory:

cd matrix-docker-ansible-deploy

This directory contains all the required configurations and playbooks for our deployment. Create all the required roles with the command:

make roles

Now we will create the variables directory:

domain_name=matrix.geeksforgeeks.org
mkdir inventory/host_vars/$domain_name

Remember to replace your domain name appropriately. Once created, copy the sample config into it:

cp examples/vars.yml inventory/host_vars/$domain_name/vars.yml

Now open the file for editing:

vim inventory/host_vars/$domain_name/vars.yml

In the opened file, we need to make a few adjustments:

##Your Domain. Example value: example.com
matrix_domain: matrix.geeksforgeeks.org

# - the `docs/configuring-playbook-IMPLEMENTATION_NAME.md` documentation page, if one is available for your implementation choice
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: ''

# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: [email protected]
matrix_ssl_lets_encrypt_support_email: ''

# The playbook creates additional Postgres users and databases (one for each enabled service)
##PostgreSQL password
devture_postgres_connection_password: 'password'

##SSL certificates retrieval
matrix_ssl_retrieval_method: manually-managed

##Retries when pulling images and creating containers 
devture_playbook_help_container_retries_delay: 9
devture_playbook_help_container_retries_count: 9

##Delay for systemd_service
devture_systemd_service_manager_up_verification_delay_seconds: 60

##Other configurations, enable them accordingly
#matrix_nginx_proxy_https_enabled: false
#matrix_coturn_enabled: false
#matrix_nginx_proxy_base_domain_serving_enabled: true

The matrix_homeserver_generic_secret_key can be generated using the command:

pwgen -s 64 1

Above are some of the basic configurations for your deployment. In the above configuration, we have set SSL retrieval to manually-managed since we want to use our own generated certificates. This can also apply in case you have custom certificates.

If this option is commented out, the playbook retrieves and auto-renews free SSL certificates from Let’s Encrypt for the domains it needs (matrix. and possibly element.)

For the case of manually-managed certificates, you need to create the SSL certificate files in the directory specified by matrix_ssl_config_dir_path. The default path is /matrix/ssl/config.

So we will create these certificates on the host we intend to do the deployment on(managed node).

For matrix.geeksforgeeks.org

sudo mkdir -p  /matrix/ssl/config/live/matrix.geeksforgeeks.org
cd /matrix/ssl/config/live/matrix.geeksforgeeks.org

While here, we will create the chain.pem and privkey.pem. Generate the self-signed certificates with the command:

sudo openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout privkey.pem \
  -addext "subjectAltName = DNS:matrix.geeksforgeeks.org" \
  -x509 -days 365 -out chain.pem

You also need the fullchain.pem which can be generated with the command:

sudo openssl req -new -x509 -key privkey.pem > fullchain.pem

Once generated, proceed and create the element.geeksforgeeks.org certificates:

sudo mkdir -p  /matrix/ssl/config/live/element.geeksforgeeks.org
cd /matrix/ssl/config/live/element.geeksforgeeks.org

Generate the certificates here:

sudo openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout privkey.pem \
  -addext "subjectAltName = DNS:element.geeksforgeeks.org" \
  -x509 -days 365 -out chain.pem

Generate the fullchain.pem certificate:

sudo openssl req -new -x509 -key privkey.pem > fullchain.pem

If you do not want to be bothered with SSL certificates, you can disable SSL certificate management in the variables file:

matrix_ssl_retrieval_method: none

But for this case, you can use an external web server which is not Nginx and no certificates will be retrieved at all. You’re free to manage them however you want.

Once the desired configurations have been made to the variables file, close it and proceed as shown below.

3. Create the Ansible Hosts Inventory file

The Ansible Hosts inventory file comprises the managed nodes. For this case, we will work with a single node.

Create the file with the command:

vim inventory/hosts

In this file, we will add the domain name, IP address and sudo user of the managed node:

[matrix_servers]
matrix.geeksforgeeks.org ansible_host=192.168.205.11 ansible_ssh_user=root

[matrix_servers:vars]
ansible_python_interpreter=/usr/bin/python3

For this case, we are using the root user for the deployment. Remember to permit root login on the remote host before you proceed:

$ sudo vim /etc/ssh/sshd_config
PermitRootLogin yes

For the changes to apply, restart the SSH service:

sudo systemctl restart ssh

Now generate the SSh keys and copy them to the managed node:

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@managed-node-IP

Now we should be able to log in to the managed node using the SSH keys. Verify if everything is okay:

$ ansible -m ping all -i inventory/hosts
matrix.geeksforgeeks.org | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

4. Installing Matrix Server using Ansible and Docker

Now we are set to perform our deployment on the remote host using the playbooks. The general command for the deployment has the syntax:

ansible-playbook -i inventory/hosts setup.yml --tags=COMMA_SEPARATED_TAGS_GO_HERE

There are several tags that can be used here:

• setup-all: runs all setup tasks including installation and uninstallation for all components, but does not start/restart services
• install-all: it is similar to setup-all, but skips uninstallation tasks. It is useful for maintaining your setup quickly when its components remain unchanged. If you adjust your vars.yml to remove components, you’d need to run setup-all though, or these components will still remain installed.
• setup-SERVICE (e.g. setup-bot-postmoogle): runs the setup tasks only for a given role, but does not start/restart services.
• install-SERVICE (e.g. install-bot-postmoogle): like setup-SERVICE, but skips uninstallation tasks.
• start – starts all systemd services and makes them start automatically in the future
• stop – stops all systemd services
• ensure-matrix-users-created: a special tag which ensures that all special users needed by the playbook (for bots, etc.) are created

There are also other tags that can be used:

• –ask-pass: applies where you don’t use SSH keys for authentication, but rather a regular password.
• –ask-become-pass: applies if you do use SSH keys for authentication, and use a non-root user to become root (sudo).

There are two ways of running the installation:

• Installing a brand new server. No data importation is required:

ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,ensure-matrix-users-created,start

• Installing a server into which you’ll import old data

ansible-playbook -i inventory/hosts setup.yml --tags=install-all

After running the above command, you can then import the existing SQLite, Postgres or media store data files from an existing installation.

Now start the services with the command:

ansible-playbook -i inventory/hosts setup.yml --tags=start

For this case, we go with the first installation type. Sample output:

Verify if the containers are running on your managed host:

# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS                        PORTS                                                                                                                                                                                                                                               NAMES
2671933d417d   nginx:1.23.2-alpine             "/docker-entrypoint.…"   About a minute ago   Up About a minute             80/tcp, 0.0.0.0:8448->8448/tcp, :::8448->8448/tcp, 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp                                                                                                                   matrix-nginx-proxy
2c836b51987c   vectorim/element-web:v1.11.15   "/docker-entrypoint.…"   About a minute ago   Up About a minute             80/tcp                                                                                                                                                                                                                                              matrix-client-element
5cb6d4e9e733   nginx:1.23.2-alpine             "/docker-entrypoint.…"   About a minute ago   Up About a minute             80/tcp                                                                                                                                                                                                                                              matrix-synapse-reverse-proxy-companion
edb82f02702a   matrixdotorg/synapse:v1.72.0    "/start.py run -m sy…"   About a minute ago   Up About a minute (healthy)   8008-8009/tcp, 8448/tcp                                                                                                                                                                                                                             matrix-synapse
5fb5ea546895   devture/exim-relay:4.95-r0-4    "/sbin/tini -- exim …"   About a minute ago   Up About a minute             8025/tcp                                                                                                                                                                                                                                            matrix-mailer
fb15e6b9f2ec   coturn/coturn:4.6.1-r0-alpine   "turnserver -c /turn…"   About a minute ago   Up About a minute             0.0.0.0:3478->3478/tcp, 0.0.0.0:3478->3478/udp, :::3478->3478/tcp, :::3478->3478/udp, 0.0.0.0:5349->5349/udp, :::5349->5349/udp, 0.0.0.0:5349->5349/tcp, 0.0.0.0:49152-49172->49152-49172/udp, :::5349->5349/tcp, :::49152-49172->49152-49172/udp   matrix-coturn
c27ce1dcd9b8   postgres:15.1-alpine            "docker-entrypoint.s…"   About a minute ago   Up About a minute             5432/tcp                                                                                                                                                                                                                                            matrix-postgres

If you make any changes to the vars.yml, either removing or adding components in the future, you need to run the installation with --tags=setup-all instead of --tags=install-all

5. Create and Register Users on the Matrix Server

After the deployment, we need to access the Matrix Server. In that case, we need to create a user to be able to log in. A user can be created using an ansible command with the below syntax:

ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=<your-username> password=<your-password> admin=<yes|no>' --tags=register-user

For example:

ansible-playbook -i inventory/hosts setup.yml --extra-vars='username=neveropen password=Passw0rd! admin=yes' --tags=register-user

Once the user has been created, proceed to the below step.

6. Access Synapse Matrix home server

Now with the right DNS records configured in /etc/hosts, access your matrix client. For this case, we have set up the element web client which is accessible on element.geeksforgeeks.org. We have a redirection configured in that when we load https://matrix.geeksforgeeks.org, it will redirect us to the element page

Sign in using the created user.

Once authenticated, you will see the below page.

Begin by creating a new room. Click on the +icon shown below.

Enter the preferred room name and make some other desired configs.

You will then have the room created, invite members and begin communication.

I will invite another user created on this server:

Once the user accepts the invitation, you can proceed with your communication. Below is a sample communication using my Matrix home server

Verdict

That marks the end of this guide. We have systematically walked through how to deploy Matrix Server using Ansible and Docker. This can make the deployment easier and faster when dealing with several hosts. I hope this was significant to you.

See more:

4 Best Open Source Alternative To Communication Proprietary Software

Install and Use FFmpeg on Rocky Linux 9 / AlmaLinux 9

Bypass Any Website Region Restrictions using Ivacy VPN