Welcome to our guide on how to setup etcd Cluster on CentOS / Ubuntu / Debian / Fedora Linux machines. This tutorial will go into detail in discussing the ideal setup of a three-node etcd cluster on a Linux box – This can be Etcd cluster on Ubuntu / Debian / CentOS / Fedora / Arch / Linux Mint or other modern Linux distribution.
etcd is a distributed and reliable key-value store for the most critical data of a distributed system. It is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.
Etcd is designed to be:
- Simple: well-defined, user-facing API (gRPC)
- Secure: automatic TLS with optional client cert authentication
- Fast: benchmarked 10,000 writes/sec
- Reliable: properly distributed using Raft
Setup Etcd Cluster CentOS / Ubuntu / Debian / Fedora Linux
This setup should work on all Linux distributions which uses systemd service manager.
This setup is based on below server network information and details.
| Short Hostname | IP Address |
|---|---|
| etcd1.mydomain.com | 192.168.18.9 |
| etcd2.mydomain.com | 192.168.18.10 |
| etcd3.mydomain.com | 192.168.18.11 |
Since all my servers uses Systemd service manager, the hostnames can be set using the commands.
# Node 1
sudo hostnamectl set-hostname etcd1.mydomain.com --static
# Node 2
sudo hostnamectl set-hostname etcd2.mydomain.com --static
# Node 3
sudo hostnamectl set-hostname etcd3.mydomain.com --staticReplace mydomain.com with your servers’ domain name. The server names can be mapped to correct IP addresses on your Local DNS or by directly adding records to /etc/hosts file on each server.
$ sudo vim /etc/hosts
192.168.18.9 etcd1.mydomain.com etcd1
192.168.18.10 etcd2.mydomain.com etcd2
192.168.18.11 etcd3.mydomain.com etcd3Step 1: Download and Install the etcd Binaries (All nodes)
Login to each etcd cluster node to be used and download etcd binaries. This is done on all nodes.
Create temporary directory.
mkdir /tmp/etcd && cd /tmp/etcdInstall wget and curl tools:
# RHEL family
sudo yum -y install wget curl
# Debian family
sudo apt -y install wget curl
# Arch/Manjaro
sudo pacman -S wget curlRemove any old versions of etcd:
rm -rf etcd*
Download etcd binary archive.
curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest \
| grep browser_download_url \
| grep linux-amd64 \
| cut -d '"' -f 4 \
| wget -i -Unarchive the file and move binaries to /usr/local/bin directory.
tar xvf etcd-v*.tar.gz
cd etcd-*/
sudo mv etcd* /usr/local/bin/
cd ..
rm -rf etcd*Check etcd and etcdctl version.
$ etcd --version
etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.5.2
API version: 3.5
$ etcdutl version
etcdutl version: 3.5.2
API version: 3.5
Step 2: Create etcd directories and user (All nodes)
We will store etcd configuration files inside the /etc/etcd directory and data in /var/lib/etcd. The user and group used to manage service is called etcd.
Create a etcd system user/group.
sudo groupadd --system etcd
sudo useradd -s /sbin/nologin --system -g etcd etcdThen create data and configurations directories for etcd.
sudo mkdir -p /var/lib/etcd/
sudo mkdir /etc/etcd
sudo chown -R etcd:etcd /var/lib/etcd/
sudo chmod -R 700 /var/lib/etcd/Step 3: Configure the etcd on all nodes
We need to populate systemd service unit files on all the three servers. But first, some environment variables are required before we can proceed.
Identify your active network interface name:
$ ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:d9:f8:f0 brd ff:ff:ff:ff:ff:ff
inet 192.168.18.9/24 brd 192.168.18.255 scope global dynamic ens3
valid_lft 56972sec preferred_lft 56972sec
inet6 fe80::f816:3eff:fed9:f8f0/64 scope link
valid_lft forever preferred_lft foreverFrom my output, the active network interface with an IP address is ens3. Etcd will be configured to run on its IP address.
Perform on all the servers
On each server, save these variables by running the commands below.
#INT_NAME="eth0"
INT_NAME="ens3" #replace with your server interface name
ETCD_HOST_IP=$(ip addr show $INT_NAME | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
ETCD_NAME=$(hostname -s)Where:
- INT_NAME is the name of your network interface to be used for cluster traffic. Change it to match your server configuration.
- ETCD_HOST_IP is the internal IP address of the specified network interface. This is used to serve client requests and communicate with etcd cluster peers.
- ETCD_NAME – Each etcd member must have a unique name within an etcd cluster. Command used will set the etcd name to match the hostname of the current compute instance.
Check variables to confirm they have correct values:
echo $INT_NAME
echo $ETCD_HOST_IP
echo $ETCD_NAMEOnce all variables are set, create the etcd.service systemd unit file:
cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
[Service]
Type=notify
User=etcd
ExecStart=/usr/local/bin/etcd \\
--name ${ETCD_NAME} \\
--data-dir=/var/lib/etcd \\
--initial-advertise-peer-urls http://${ETCD_HOST_IP}:2380 \\
--listen-peer-urls http://${ETCD_HOST_IP}:2380 \\
--listen-client-urls http://${ETCD_HOST_IP}:2379,http://127.0.0.1:2379 \\
--advertise-client-urls http://${ETCD_HOST_IP}:2379 \\
--initial-cluster-token etcd-cluster-0 \\
--initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 \\
--initial-cluster-state new \
[Install]
WantedBy=multi-user.target
EOFIf you don’t have working name resolution or mappings added to /etc/hosts file, then replace etcd1, etcd2 and etcd3 with your nodes IP addresses.
For CentOS / RHEL Linux distributions, set SELinux mode to permissive.
sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/configIf you have active firewall service, allow ports 2379 and 2380.
# RHEL / CentOS / Fedora firewalld
sudo firewall-cmd --add-port={2379,2380}/tcp --permanent
sudo firewall-cmd --reload
# Ubuntu/Debian
sudo ufw allow proto tcp from any to any port 2379,2380Step 4: Start the etcd Server
Start etcd service by running the following commands on each cluster node.
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcdConfirm that etcd service is running on all nodes.
[neveropen@etcd1 ~]$ systemctl status etcd -l
● etcd.service - etcd service
Loaded: loaded (/etc/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2019-06-03 18:20:49 UTC; 30s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 5931 (etcd)
CGroup: /system.slice/etcd.service
└─5931 /usr/local/bin/etcd --name etcd1 --data-dir=/var/lib/etcd --initial-advertise-peer-urls http://192.168.18.9:2380 --listen-peer-urls http://192.168.18.9:2380 --listen-client-urls http://192.168.18.9:2379,http://127.0.0.1:2379 --advertise-client-urls http://192.168.18.9:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --initial-cluster-state new
....................................................................................
[neveropen@etcd2 ~]$ systemctl status etcd -l
● etcd.service - etcd service
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-06-03 18:20:49 UTC; 2min 17s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 5949 (etcd)
CGroup: /system.slice/etcd.service
└─5949 /usr/local/bin/etcd --name etcd2 --data-dir=/var/lib/etcd --initial-advertise-peer-urls http://192.168.18.10:2380 --listen-peer-urls http://192.168.18.10:2380 --listen-client-urls http://192.168.18.10:2379,http://127.0.0.1:2379 --advertise-client-urls http://192.168.18.10:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --initial-cluster-state new
....................................................................................
[neveropen@etcd3 ~]$ systemctl status etcd -l
● etcd.service - etcd service
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-06-03 18:20:49 UTC; 3min 20s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 5974 (etcd)
CGroup: /system.slice/etcd.service
└─5974 /usr/local/bin/etcd --name etcd3 --data-dir=/var/lib/etcd --initial-advertise-peer-urls http://192.168.18.11:2380 --listen-peer-urls http://192.168.18.11:2380 --listen-client-urls http://192.168.18.11:2379,http://127.0.0.1:2379 --advertise-client-urls http://192.168.18.11:2379 --initial-cluster-token etcd-cluster-0 --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --initial-cluster-state newStep 5: Test Etcd Cluster installation
Test your setup by listing the etcd cluster members:
$ etcdctl member list
152d6f8123c6ac97: name=etcd3 peerURLs=http://etcd3:2380 clientURLs=http://192.168.18.11:2379 isLeader=false
332a8a315e569778: name=etcd2 peerURLs=http://etcd2:2380 clientURLs=http://192.168.18.10:2379 isLeader=false
aebb404b9385ccd4: name=etcd1 peerURLs=http://etcd1:2380 clientURLs=http://192.168.18.9:2379 isLeader=trueTo use etcd v3, you need to explicitly specify version.
$ ETCDCTL_API=3 etcdctl member list
152d6f8123c6ac97, started, etcd3, http://etcd3:2380, http://192.168.18.11:2379
332a8a315e569778, started, etcd2, http://etcd2:2380, http://192.168.18.10:2379
aebb404b9385ccd4, started, etcd1, http://etcd1:2380, http://192.168.18.9:2379Also check cluster health by running the command:
$ etcdctl endpoint healthLet’s also try writing to etcd.
$ etcdctl put /message "Hello World"
OK
$ etcdctl put foo "Hello World!"
OKRead the value of message back – It should work on all nodes.
$ etcdctl get /message
/message
Hello World
$ etcdctl get foo
foo
Hello World!Step 6 – Test Leader failure
When a leader fails, the etcd cluster automatically elects a new leader. The election does not happen instantly once the leader fails. It takes about an election timeout to elect a new leader since the failure detection model is timeout based.
During the leader election, the cluster cannot process any writes. Write requests sent during the election are queued for processing until a new leader is elected.
Our current leader is etcd1 – Node 1.
$ etcdctl member list
152d6f8123c6ac97: name=etcd3 peerURLs=http://etcd3:2380 clientURLs=http://192.168.18.11:2379 isLeader=false
332a8a315e569778: name=etcd2 peerURLs=http://etcd2:2380 clientURLs=http://192.168.18.10:2379 isLeader=false
aebb404b9385ccd4: name=etcd1 peerURLs=http://etcd1:2380 clientURLs=http://192.168.18.9:2379 isLeader=trueLet’s take it down.
[neveropen@etcd1 ~]$ sudo systemctl stop etcdCheck new Leader – Now etcd2 server.
$ etcdctl member list
152d6f8123c6ac97: name=etcd3 peerURLs=http://etcd3:2380 clientURLs=http://192.168.18.11:2379 isLeader=false
332a8a315e569778: name=etcd2 peerURLs=http://etcd2:2380 clientURLs=http://192.168.18.10:2379 isLeader=true
aebb404b9385ccd4: name=etcd1 peerURLs=http://etcd1:2380 clientURLs=http://192.168.18.9:2379 isLeader=falseOnce etcd1 service is started, the leader will remain etcd2 unless it goes down.
Conclusion
You now a working three node Etcd cluster installed on CentOS / Ubuntu / Debian Linux system. Visit Etcd documentation for detailed setup and usage guide.
Similar setups:
