Install Prometheus with Node Exporter on Rocky Linux 9/AlmaLinux 9

Today, Linux is one of the widely used operating systems, powering a large part of the internet applications. Almost half of the top-ranked websites in the world run on Linux. This is because Linux offers customizability, low-resource requirements, security, reliability, and continuous development by its community,

When running applications on on-premise and cloud infrastructures, it is important to have high performance and availability. Fortunately, there are quite a number of tools that can help you monitor the performance and health of the Linux servers.

Monitoring tools are divided into the following categories:

• Command Line Tools – such as atop, top, htyop, apachetop e.t.c
• Desktop Monitoring Tools – for example ntopng, iftop, bandwidth e.t.c
• Infrastructure Monitoring Tools – such as OpenNMS, SysUsage, Nagios, Cacti, Zabbix e.t.c
• Log Monitoring Tools – such as Logwatch, Swatch, MultiTail e.t.c
• Network Monitoring Tools – they include strace, DTrace, webmin, collected e.t.c

In this guide, we will walk through how to install Prometheus with Node Exporter on Rocky Linux 9 / AlmaLinux 9

Prometheus is a free and open-source toolkit used for time series event monitoring and alerting. It was originally developed by SoundCloud in 2012. Prometheus was then adopted by several companies and active developers around the world. In 2016, it was elevated to the Cloud Native Computing Foundation as the second hosted project after Kubernetes.

Prometheus is preferred due to the following features:

• Time-series metrics collection which happens via a pull model over HTTP
• It uses PromQL, which is a flexible query language to leverage this dimensionality
• It doesn’t rely on distributed storage; single server nodes are autonomous
• It uses a multi-dimensional data model where time series data is identified by metric name and key/value pairs
• Targets are discovered via service discovery or static configuration
• It supports multiple modes of graphing and dashboarding.

Prometheus works by collecting and storing metrics as time series data. This means that the metrics are stored with the timestamp at which they were collected. Metrics are usually numeric measurements that vary from time to time. They may be different from one application to the other. For example, on web servers, you might be interested in request times, for database servers, it might be the number of active queries/connections.

The Prometheus ecosystem encompasses several components. They include:

• Prometheus server – scrapes and stores time series data
• Client libraries – used to instrument application code
• Push gateway – to support short-lived jobs
• Special-purpose exporters – for services such as HAProxy, StatsD, Graphite e.t.c
• Alert manager – for alert handling

The diagram below can be used to illustrate the Prometheus Architecture

For Prometheus to collect the metrics, you need to install the ‘exporter‘ application to expose the data and metrics. Node exporter is used to expose the hardware and kernel-related metrics on Linux systems.

Let’s dive in!

1. Add Prometheus Repositories

Prometheus is not provided in the default Rocky Linux 9 / AlmaLinux 9 repositories. The repositories need to be added to the system to be able to install Prometheus.

Begin by enabling the EPEL repository:

sudo dnf -y install epel-release vim

Add repository into the system.

curl -s https://packagecloud.io/install/repositories/prometheus-rpm/release/script.rpm.sh | sudo bash

2. Install and Configure Prometheus Server

Once the repository has been added, Prometheus can be installed with the command:

sudo dnf install prometheus -y

Dependency Tree:

....
Transaction Summary
============================================================================================================================================
Install  1 Package

Total download size: 35 M
Installed size: 166 M
Is this ok [y/N]: y

Once installed, Prometheus stores its data at /var/lib/prometheus and config files at /etc/prometheus/.

The default configuration file is at /etc/prometheus/prometheus.yml

sudo vim /etc/prometheus/prometheus.yml

It can be customized as desired.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
.........
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["192.168.205.13:9090"]

Make sure that the file fits your initial settings for Prometheus. Save it, start and enable the service with the command:

sudo systemctl start prometheus
sudo systemctl enable prometheus

Check the status of the service:

$ systemctl status prometheus.service 
● prometheus.service - The Prometheus monitoring system and time series database.
     Loaded: loaded (/usr/lib/systemd/system/prometheus.service; enabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-08-14 11:51:18 CEST; 7s ago
       Docs: https://prometheus.io
   Main PID: 2111 (prometheus)
      Tasks: 6 (limit: 23441)
     Memory: 19.6M
        CPU: 46ms
     CGroup: /system.slice/prometheus.service
             └─2111 /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data --web.con

Allow the port through the firewall:

sudo firewall-cmd --add-port=9090/tcp --permanent
sudo firewall-cmd --reload

At this time, you can access Prometheus using the URL http://IP_Address:9090

3. Install and Configure Node Exporter

You have two options of installing Node Exporter.

Option 1: Install from RPM Repository (Recommended)

Add Prometheus repository as covered in step 1, then proceed to install the package from it.

sudo dnf -y install node_exporter

Start and enable the service after installation:

sudo systemctl enable --now node_exporter

Option 2: Download and install manually

Install curl and wget packages

sudo dnf -y install wget tar curl vim

Node exporter can be downloaded from the Github Release page. You can also pull the archive using wget as shown:

##For AMD64
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-amd64.tar.gz

##For ARM64
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-arm64.tar.gz

##For i386
VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/node_exporter/releases/download/v$VERSION/node_exporter-$VERSION.linux-386.tar.gz

Once downloaded, extract the archive:

tar -xf node_exporter-$VERSION.linux-*.tar.gz

Move the binary file to your PATH:

sudo mv node_exporter-*/node_exporter /usr/local/bin

To run node exporter as a service, create a dedicated user:

sudo adduser -M -r -s /sbin/nologin prometheus

Create the systemd service file:

sudo vim /etc/systemd/system/node_exporter.service

Add the below lines to the file:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Save the file and configure SELinux for the Path to be accessible

sudo /sbin/restorecon -v /usr/local/bin/node_exporter

and reload the system daemon:

sudo systemctl daemon-reload

Start and enable Node exporter;

sudo systemctl enable --now node_exporter

Verify if the service is running:

$ systemctl status node_exporter
● node_exporter.service - Node Exporter
     Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-08-14 12:00:27 CEST; 4s ago
   Main PID: 2401 (node_exporter)
      Tasks: 3 (limit: 23441)
     Memory: 2.8M
        CPU: 8ms
     CGroup: /system.slice/node_exporter.service
             └─2401 /usr/local/bin/node_exporter

Check the port on which the service is listening:

$ sudo ss -aplnt | grep node
LISTEN 0      4096               *:9100            *:*    users:(("node_exporter",pid=2401,fd=3))

Allow this port through the firewall.

sudo firewall-cmd --add-port=9100/tcp --permanent
sudo firewall-cmd --reload

Now configure Prometheus to use node exporter.

sudo vim /etc/prometheus/prometheus.yml

Under the scrape_config section, add the new job for Node exporter:

  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['SERVER-IP:9100']

Remember to replace the server IP appropriately. Save the file and restart the Prometheus service

sudo systemctl restart prometheus

4. Install and Configure Blackbox exporter

The Blackbox exporter enables probing of endpoints over HTTP, HTTPS, DNS, TCP, and ICMP. You can install from YUM repository or manually.

Option 1) Install Blackbox exporter from Prometheus repository

Add the repository as demonstrated in step 1 and install the package:

sudo dnf -y install blackbox_exporter

The settings of Blackbox exporter are in the file /etc/prometheus/blackbox.yml. You can modify or just start and enable the service:

sudo systemctl enable --now blackbox_exporter

Enable required port in the firewall:

sudo firewall-cmd --add-port=9115/tcp --permanent
sudo firewall-cmd --reload

Option 2) Install Blackbox exporter manually

The exporter can be downloaded from the Github Release page. But first install curl and wget packages

sudo dnf -y install wget tar curl vim

You can also pull the file as shown:

##For AMD64
VERSION=$(curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/blackbox_exporter/releases/download/v$VERSION/blackbox_exporter-$VERSION.linux-amd64.tar.gz

##For ARM64
VERSION=$(curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/blackbox_exporter/releases/download/v$VERSION/blackbox_exporter-$VERSION.linux-arm64.tar.gz

##For i386
VERSION=$(curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/prometheus/blackbox_exporter/releases/download/v$VERSION/blackbox_exporter-$VERSION.linux-386.tar.gz

Once downloaded, extract the archive:

tar -xf blackbox_exporter-$VERSION.linux-*.tar.gz

Now move the binary file to your PATH.

sudo mv blackbox_exporter-*/blackbox_exporter /usr/local/bin

Create a system user for the Blackbox exporter.

sudo adduser -M -r -s /sbin/nologin blackbox_exporter

Create the config file with the correct ownership.

sudo mkdir /etc/blackbox_exporter
sudo mv  blackbox_exporter-*/blackbox.yml /etc/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml

Create a systemd service file for Blackbox exporter.

sudo vim /etc/systemd/system/blackbox_exporter.service

Add the below lines:

[Unit]
Description=Blackbox Exporter
After=network.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

Save the file and configure SELinux for the Path to be accessible

sudo /sbin/restorecon -v /usr/local/bin/blackbox_exporter

Reload the system daemon:

sudo systemctl daemon-reload

Start and enable the service:

sudo systemctl enable --now blackbox_exporter

Verify if the service is running as desired:

$ systemctl status blackbox_exporter
● blackbox_exporter.service - Blackbox Exporter
     Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-08-14 12:16:30 CEST; 5s ago
   Main PID: 2752 (blackbox_export)
      Tasks: 6 (limit: 23441)
     Memory: 6.8M
        CPU: 7ms
     CGroup: /system.slice/blackbox_exporter.service
             └─2752 /usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

Verify the Blackbox exporter port:

$ sudo ss -aplnt | grep black
LISTEN 0      4096               *:9115            *:*    users:(("blackbox_export",pid=2752,fd=3))

Allow this port through the firewall.

sudo firewall-cmd --add-port=9115/tcp --permanent
sudo firewall-cmd --reload

Configuring Prometheus for Black exporter

Configure Prometheus for Blackbox exporter.

sudo vim /etc/prometheus/prometheus.yml

Blackbox exporter supports endpoints probing over HTTP, HTTPS, DNS, TCP, and ICMP. we will create a simple HTTP job for Blackbox as shown:

scrape_configs:
.......
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://192.168.205.13:9090    # Target to probe with http.
        - https://prometheus.example.com   # Target to probe with https.
        - http://example.com:8080 # Target to probe with http on port 8080.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: ServerIP:9115

In the above file, we are monitoring the Prometheus web as our web endpoint. Replace the Server IP as preferred.

See other examples below:

scrape_configs:
......
  # Using [icmp] module
  # job_name can be anything
  - job_name: 'Blackbox_icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        #hostname or IP address of target Host
        - ServerIP
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # Blackbox exporter Host:Port
        replacement: exporterserverip:9115

  # Using [ssh_banner] module
  - job_name: 'Blackbox_ssh'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    static_configs:
      - targets:
        - ServerIP:22
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: exporterserverip:9115

  # Using [tcp_connect] module
  - job_name: 'Blackbox_tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        # Target example is MariaDB/MySQL
        - ServerIP:3306
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: exporterserverip:9115

Save the file and restart Prometheus:

sudo systemctl restart prometheus

5. Dashboard access and Metrics viewing on Prometheus

Once the exporters have been installed and configured, you can view dashboards on Prometheus. Navigate to Status->Targets

There are 3 different endpoints, Prometheus, Node exporter, and Blackbox exporter.

Click show more to view the job details.

Now click on Graphs and set up the desired Graphs on Prometheus. To get graphs, type a PromQL query such as node_os_info and execute it.

You can also view network speeds using the query node_network_speed_bytes and navigate to the graphs tab

To visualize the graphs better, you can install Grafana as illustrated in the guide below:

• Install Grafana 9 with Prometheus on Rocky Linux 9 / Alma Linux 9

That marks the end of this guide on how to install Prometheus with Node Exporter on Rocky Linux 9 / AlmaLinux 9. We have also installed and configured the Blackbox exporter.

See more on this page:

Secure Prometheus Server With Basic Password Authentication

How to Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes

How to Monitor BIND DNS server with Prometheus and Grafana