Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes

2 September 2024

0

Prometheus is an open source monitoring solution that stores all its data in a time series database. Prometheus has a multi-dimensional data-model and a powerful query language that is used to generate reports of the resources being monitored. This tutorial explains how to monitor a Linux server performance with Prometheus and Grafana.

Prometheus node exporter exports hardware and OS metrics exposed by *NIX kernels for consumption by Prometheus. This exporter is written in Go with pluggable metric collectors.

Similar Prometheus articles available on this blog are:

Setup Procedure:

Install Prometheus and Grafana
Install Prometheus Node Exporter on Linux servers to be monitored
Configure Node Exporter
Configure Prometheus server with Scrap jobs
Add Dashboards to Grafana
Start visualizing system metrics on Grafana

For installation of Prometheus and Grafana use:

Step 1: Add Prometheus system user

We’ll add a user account to run nod exporter service. It is safe since it doesn’t have access to the interactive shell and home directory.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

We added a system user called prometheus whose default group is prometheus.

Step 2: Install Prometheus Node Exporter

Install wget and curl

### RHEL based systems ###
sudo yum -y install wget curl vim

### Debian based systems ###
sudo apt update
sudo apt install curl wget vim -y

Next is the latest release of Prometheus Node exporter:

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest | grep browser_download_url | grep linux-amd64 |  cut -d '"' -f 4 | wget -qi -

Extract the file downloaded:

tar xvf node_exporter-*linux-amd64.tar.gz

Move the downloaded file to the /usr/local/bin directory:

cd node_exporter*/
sudo mv node_exporter /usr/local/bin/
cd ~/

The version installed can be confirmed using the command:

$ node_exporter  --version
node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)
  build user:       root@6e7732a7b81b
  build date:       20221129-18:59:09
  go version:       go1.19.3
  platform:         linux/amd64

Step 3: Configure Node Exporter systemd / Init script

Collectors are enabled by providing a --collector.<name> flag.

Collectors that are enabled by default can be disabled by providing a --no-collector.<name>flag.

sudo vim /etc/systemd/system/node_exporter.service

Add below content:

[Unit]
Description=Prometheus
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/node_exporter \
    --collector.cpu \
    --collector.diskstats \
    --collector.filesystem \
    --collector.loadavg \
    --collector.meminfo \
    --collector.filefd \
    --collector.netdev \
    --collector.stat \
    --collector.netstat \
    --collector.systemd \
    --collector.uname \
    --collector.vmstat \
    --collector.time \
    --collector.mdadm \
    --collector.zfs \
    --collector.tcpstat \
    --collector.bonding \
    --collector.hwmon \
    --collector.arp \
    --web.listen-address=:9100 \
    --web.telemetry-path="/metrics"

Start the service and enable it to start on boot:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Step 4: Configure firewall

If you have an active firewall on your server, e.g firewalld, ufw, open port 9100

sudo ufw allow 9100

For CentOS 7 system, use firewalld:

sudo firewall-cmd --add-port=9100/tcp --permanent
sudo firewall-cmd --reload

For Init Linux system like CentOS 6, you can use daemonize to start the service in the background.

Install daemonize:

sudo yum install daemonize
sudo apt install daemonize

Once installed, create node_exporter init script:

sudo vim /etc/init.d/node_exporter

Add below script:

#!/bin/bash
# Author: Josphat Mutai, [email protected] , https://github.com/jmutai
# node_exporter     This shell script takes care of starting and stopping Prometheus apache exporter
#
# chkconfig: 2345 80 80
# description: Prometheus apache exporter  start script
# processname: node_exporter
# pidfile: /var/run/node_exporter.pid

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0
PROGNAME=node_exporter
PROG=/usr/local/bin/${PROGNAME}
RUNAS=prometheus
LOCKFILE=/var/lock/subsys/${PROGNAME}
PIDFILE=/var/run/${PROGNAME}.pid
LOGFILE=/var/log/${PROGNAME}.log
DAEMON_SYSCONFIG=/etc/sysconfig/${PROGNAME}

# GO CPU core Limit

#GOMAXPROCS=$(grep -c ^processor /proc/cpuinfo)
GOMAXPROCS=1

# Source config

. ${DAEMON_SYSCONFIG}

start() {
    if [[ -f $PIDFILE ]] > /dev/null; then
        echo "node_exporter  is already running"
        exit 0
    fi

    echo -n "Starting node_exporter  service…"
    daemonize -u ${USER} -p ${PIDFILE} -l ${LOCKFILE} -a -e ${LOGFILE} -o ${LOGFILE} ${PROG} ${ARGS}
    RETVAL=$?
    echo ""
    return $RETVAL
}

stop() {
    if [ ! -f "$PIDFILE" ] || ! kill -0 $(cat "$PIDFILE"); then
        echo "Service not running"
        return 1
    fi
    echo 'Stopping service…'
    #kill -15 $(cat "$PIDFILE") && rm -f "$PIDFILE"
    killproc -p ${PIDFILE} -d 10 ${PROG}
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f ${LOCKFILE} ${PIDFILE}
    return $RETVAL
}

status() {
    if [ -f "$PIDFILE" ] || kill -0 $(cat "$PIDFILE"); then
      echo "apache exporter  service running..."
      echo "Service PID: `cat $PIDFILE`"
    else
      echo "Service not running"
    fi
     RETVAL=$?
     return $RETVAL
}

# Call function
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    status)
        status
        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 2
esac

Create Arguments configuration file:

sudo vim /etc/sysconfig/node_exporter

Add:

"--collector.cpu \
--collector.diskstats \
--collector.filesystem \
--collector.loadavg \
--collector.meminfo \
--collector.filefd \
--collector.netdev \
--collector.stat \
--collector.netstat \
--collector.systemd \
--collector.uname \
--collector.vmstat \
--collector.time \
--collector.mdadm \
--collector.xfs \
--collector.zfs \
--collector.tcpstat \
--collector.bonding \
--collector.hwmon \
--collector.arp \
--web.listen-address=:9100

Test the script:

# /etc/init.d/node_exporter
Usage: /etc/init.d/node_exporter {start|stop|restart}

Step 5: Start the Prometheus node exporter service

For systemd, start using:

sudo systemctl start node_exporter
sudo systemctl enable node_exporter

For Init system use:

sudo /etc/init.d/node_exporter start
sudo chkconfig node_exporter on

You can verify using:

$ sudo /etc/init.d/node_exporter status
apache exporter  service running...
Service PID: 1970

$ sudo chkconfig --list | grep node_exporter
node_exporter 0:off   1:off   2:on    3:on    4:on    5:on    6:off

$ sudo ss -tunelp | grep 9100
tcp    LISTEN     0      128      :::9100                 :::*                   users:(("node_exporter",pid=16105,fd=3)) uid:997 ino:193468 sk:ffff8a0a76f52a80 v6only:0 <->

Step 6: Add exporter job to Prometheus

The second last step is to add a job to the Prometheus server for scraping metrics. Edit /etc/prometheus/prometheus.yml

$ sudo vim /etc/prometheus/prometheus.yml
# Linux Servers
  - job_name: apache-linux-server1
    static_configs:
      - targets: ['10.1.10.20:9100']
        labels:
          alias: server1

  - job_name: apache-linux-server2
    static_configs:
      - targets: ['10.1.10.21:9100']
        labels:
          alias: server2

Restart prometheus service for scraping to start

sudo systemctl restart prometheus

Test access to port 9100 from Prometheus server

$ telnet 10.1.10.20 9100
Trying 10.1.10.20...
Connected to 10.1.10.20.
Escape character is '^]'.
^]

Step 7: Add Dashboard to Grafana

You can create your own Grafana dashboard or import from a collection of community shared dashboards. Below is a list of dashboards than has been created to show classical system metrics of your *NIX server.

Recommended:

For demo purposes, we’ll use the first dashboard with ID 1860.

Add Prometheus data source

With Prometheus data source added to Grafana, Import Apache Grafana Dashboard by navigating to Dashboard > Import. Use 1860 for Grafana Dashboard ID.