After initial setup of a Kubernetes cluster, the most common day 2 operation is scaling your cluster up by adding more nodes that runs your workloads – containers and pods. The way you scale your cluster depends on the tools that were used initially during cluster bootstrapping. This guide will demonstrate how you can add more Worker nodes to Kubernetes cluster using kubeadm command line tool.
See below guides for Kubernetes installation with kubeadm:
- Deploy Kubernetes Cluster on CentOS 7/8 With Ansible
- Install Kubernetes Cluster on Rocky Linux 8 with Kubeadm & CRI-O
- Install Kubernetes Cluster on Ubuntu with kubeadm
Before you begin
- You need to have a working Kubernetes cluster – configured and working control plane node
- Container runtime (Docker,cri-o,containerd, e.t.c) and Kubernetes tools(kubeadm and kubelet) installed in your Worker node.
- If using Firewall such as firewalld, ports 10250, 30000-32767 and ports required by your Pod network add-on should be opened in the firewall.
- SSH access to the machine to be added
- Configured kubectl for checking if the node is available in your cluster
These standard requirements are covered in detail in our Kubernetes cluster installations.
Here are the standard steps you’ll follow to join or add a new Worker Node to an existing Kubernetes Cluster.
Step 1: Get join Token
A token is required when joining a new worker node to the Kubernetes cluster. When you bootstrap a cluster with kubeadm, a token is generated which expires after 24 hours.
Check if you have a token – Run the command on Control node:
$ kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
nx1jjq.u42y27ip3bhmj8vj 21h 2020-01-10T20:33:08Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-tokenIf the token is expired, generate a new one with the command:
sudo kubeadm token createThe grab the token generated using:
kubeadm token list
You can also generate token and print the join command:
kubeadm token create --print-join-commandStep 2: Get Discovery Token CA cert Hash
For token-based discovery, the join command validates that the root CA public key by matching its hash with the one provided. Get the discovery token CA cert hash on the control node by running the command below.
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'Step 3: Get API Server Advertise address
Get your API server endpoint with kubectl command:
$ kubectl cluster-info
Kubernetes master is running at https://192.168.122.195:6443
KubeDNS is running at https://192.168.122.195:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.122.195:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
As shown in the output, my API is on https://192.168.122.195:6443.
Step 4: Join new Kubernetes Worker Node a Cluster
The kubeadm join command is used to bootstrap a Kubernetes worker node or an additional control plane node, and join it to the cluster. The command syntax for joining a worker node to cluster is:
kubeadm join [api-server-endpoint] [flags]The common flags required are:
--tokenstring: Token to use--discovery-token-ca-cert-hash: Has a format: <type>:<value>
So our complete join command will have the format:
kubeadm join \
<control-plane-host>:<control-plane-port> \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash>Example:
sudo kubeadm join \
192.168.122.195:6443 \
--token nx1jjq.u42y27ip3bhmj8vj \
--discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6cCommand output:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.Wait for the node to have status “Ready” – Check on control node
$ watch kubectl get nodes
The process may take few minutes as container images are pulled before services are configured and brought up.
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
491662a7affc k8s.gcr.io/kube-proxy "/usr/local/bin/kube…" About a minute ago Up About a minute k8s_kube-proxy_kube-proxy-zt8f9_kube-system_8f7f407c-d415-4ae0-8066-fbdf3dcb1570_0
743e7239b295 k8s.gcr.io/pause:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_calico-node-rr666_kube-system_0c5a592c-08df-4303-a50d-0d93a2aec0ce_0
a69aad86ba4a k8s.gcr.io/pause:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_kube-proxy-zt8f9_kube-system_8f7f407c-d415-4ae0-8066-fbdf3dcb1570_0Step 5: Removing a Worker Node from the Cluster
To remove a Kubernetes worker node from the cluster, perform the following operations.
- Migrate pods from the node:
kubectl drain <node-name> --delete-local-data --ignore-daemonsets- Prevent a node from scheduling new pods use – Mark node as unschedulable
kubectl cordon <node-name>- Revert changes made to the node by ‘kubeadm join‘ – Run on worker node to be removed
sudo kubeadm reset- You can then redo the same process of joining a new node to the cluster once the kubeadm reset command has been executed successfully.
Persistent Volume storage solutions for Kubernetes:
Also check:
