How To Monitor Etcd Cluster with Prometheus and Grafana

5 September 2024

1

Monitoring is the next imperative duty after setting up any service used in Production. Having proper monitoring for your Etcd cluster ensures you can respond to issues quickly and know when it’s time to scale. There are a plethora of tools that can be used to monitor an etcd cluster. This guide will discuss the use of Prometheus and Grafana Dashboards, with optional alerting that can be configured.

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging. Test by running:

$ curl -L http://localhost:2379/metrics

Before you proceed, I expect you to have a ready Etcd cluster on a Linux server. We have a detailed guide on how to setup an Etcd cluster on Linux -CentOS/Ubuntu e.t.c. With a running cluster, you can proceed to setup Prometheus and Grafana to be used in monitoring.

Step 1: Install Grafana

You need Grafana Data visualization & Monitoring tool installed on a Linux system. We have a number of guides you can refer to.

Install Grafana on Ubuntu / CentOS / Fedora.

Step 2: Install Prometheus

Prometheus and Grafana can co-exist on same server. For the monitoring to work, the Prometheus server should be able to access Etcd nodes on the service ports.

Check our recent guide which covers the installation of Prometheus on CentOS & Ubuntu Linux systems.

Step 3: Configure Prometheus

We need to set Prometheus’s scraper to target the etcd cluster endpoints:

sudo vim /etc/prometheus/prometheus.yml

Add like below.

- job_name: 'etcd'
    static_configs:
    - targets: ['192.168.18.9:2379','192.168.18.10:2379','192.168.18.11:2379'']

Replace the IP addresses specified with your Etcd cluster node’s IP addresses. Restart Prometheus service after adding the configuration lines to the file.