You want to deploy a Lightweight Kubernetes Cluster with ease and less memory footprint?, all in a binary less than 40mb?. Kubernetes has been a game changer in how containerized workloads are deployed and managed at immense scale. The main challenge for Developers revolve around setup process and resource requirements to have a working Kubernetes Cluster.
For Production deployment refer to:
- Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray
- Install Production Kubernetes Cluster with Rancher RKE
For semi manual deployment with Kubeadm, check:
For Development and test impetus, a user should be able to deploy Kubernetes with least resource utilization and low hardware specs. K3s is a tiny Kubernetes distribution created and optimized by Rancher team to run on IoT and ARM powered devices.
Since K3s is optimized to use less resources, some Kubernetes features are stripped off. These include:
- Legacy, alpha, non-default features
- Most in-tree plugins (cloud providers and storage plugins) which can be replaced with out of tree addons.
- etcd3 replaced by sqlite3 as the default storage mechanism
Deploy Lightweight Kubernetes Cluster in 5 minutes with K3s
For my deployment, I have three servers running Debian 10 each with 1GB of Ram and 1vcpu. One of the servers will be used as master and other two as worker nodes.
$ openstack server list
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| 4df6a6dc-26e8-4ae0-8b6e-2f97daec0ef3 | k3s-master | ACTIVE | private=10.10.1.159 | Debian-10 | m1.tiny |
| 5ca13239-b745-4f62-ab11-0a27949c9b35 | k3s-node02 | ACTIVE | private=10.10.1.142 | Debian-10 | m1.tiny |
| a54997f2-4d94-4718-86ab-73609b328761 | k3s-node01 | ACTIVE | private=10.10.1.126 | Debian-10 | m1.tiny |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+I’ll add A records for the servers on /etc/hosts file on each server.
$ sudo vim /etc/hosts
10.10.1.159 k3s-master
10.10.1.126 k3s-node01
10.10.1.142 k3s-node02
Install K3s on Master node
There are many ways to run k3s. The quickest method is installation via provided bash script. This script provides a convenient way for installing to systemd or openrc.
curl -sfL https://get.k3s.io | sh -Installation output.
[INFO] Finding release for channel stable
[INFO] Using v1.27.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
The service is automatically started after installation.
$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-03-08 11:56:10 UTC; 16s ago
Docs: https://k3s.io
Process: 1412 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 1414 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 1416 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 1421 (k3s-server)
Tasks: 56
Memory: 544.0M
CPU: 22.008s
CGroup: /system.slice/k3s.service
├─1421 "/usr/local/bin/k3s server"
└─1466 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containe>
Mar 08 11:56:24 jammy k3s[1421]: I0308 11:56:24.641228 1421 iptables.go:270] bootstrap done
Mar 08 11:56:25 jammy k3s[1421]: W0308 11:56:25.294598 1421 handler_proxy.go:105] no RequestInfo found in the context
Mar 08 11:56:25 jammy k3s[1421]: W0308 11:56:25.294635 1421 handler_proxy.go:105] no RequestInfo found in the context
Mar 08 11:56:25 jammy k3s[1421]: E0308 11:56:25.295828 1421 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIS>
Mar 08 11:56:25 jammy k3s[1421]: I0308 11:56:25.295900 1421 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 08 11:56:25 jammy k3s[1421]: E0308 11:56:25.295963 1421 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: Respons>
Mar 08 11:56:25 jammy k3s[1421]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Mar 08 11:56:25 jammy k3s[1421]: I0308 11:56:25.297181 1421 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 08 11:56:25 jammy k3s[1421]: E0308 11:56:25.771986 1421 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameser>
Mar 08 11:56:25 jammy k3s[1421]: I0308 11:56:25.922767 1421 request.go:690] Waited for 1.216625763s due to client-side throttling, not priority and fairness, request: POST:https://127.0.0.1:6443>
A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml:
$ sudo cat /etc/rancher/k3s/k3s.yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTmpJME5qazJPREl3SGhjTk1qSXdPVEEyTVRNd09EQXlXaGNOTXpJd09UQXpNVE13T0RBeQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTmpJME5qazJPREl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUQldzaU5RNHpUZU1Dc3pxd2hzWTRDdExnc0V3WE9MajJ4VEtHRlNjYS8KWVdYM3hyNFp0TTFJcENXSXAwNjlMcDRtWUo5MUlyWm8zSDdCQmtaQzU5TXNvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWNLekhBYkd3cXJoZVFOQzZpdG56CnVYNGJ3bHd3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQU9qOVBlTkNOaXdvcHhPc2N6Rkgzd1hUT2dHdDl5eUcKR0RKNmR5bXljM29HQWlBTjJXUUtSQ2MzNkVyY0NacS8rMHJZcGEraEJKZVFVbGtBcFZ1bThFcVFLQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrRENDQVRlZ0F3SUJBZ0lJVXVpb0JCVDRSVGt3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOall5TkRZNU5qZ3lNQjRYRFRJeU1Ea3dOakV6TURnd01sb1hEVEl6TURrdwpOakV6TURnd01sb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJMT0NUdSs1eHBHRjlqMmcKeWVGK3REYnZKejg2M29wbmtFVUVQL0VOTDV4SEtINVpkQlNpM0pHa1AyTzVNUWNxcU9lOEdMd01lTVpHdG9jNAp2WU9ybWV5alNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUVJyanUvVnlvRHg4ek9tWmdVL094UWYwUi8yekFLQmdncWhrak9QUVFEQWdOSEFEQkUKQWlBenlDR29vTERvem5hNVZ0ODB6UnMyT2hkT05KbE5VVnRwWWh4N0RNN0RSQUlnUUozVHZhRitZaGxBY2FjTQpYV1VPNmpmMUZnZkw2MU1qb0VUeGFVbzk5Szg9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyTmpJME5qazJPREl3SGhjTk1qSXdPVEEyTVRNd09EQXlXaGNOTXpJd09UQXpNVE13T0RBeQpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyTmpJME5qazJPREl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRc2VQOERKZC96dVpPb0lHazlOUllJR2lmNURIRWhESVp1ZGkvc0RobmYKTkFRNEZ6cUl4U0RaZlFHdEw1UTErU0QzOXBXb1puYTh2aHJPYzZrcVAvbU1vMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUVhNDd2MWNxQThmTXpwbVlGUHpzClVIOUVmOXN3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQVBmQkZxcUN5SlI1YnRIVW4xS3JMKzZOYkwyYXJ3VmwKUGt4alNKVlpOUUE5QWlCQ2RoVHVuL2d1NWw0WHVSSE9Ba2xWb0E3K1ZNVGwvdG05OUZXSEk1M3J4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUp2MEE5UjUramtDRi82RW9NWXZadU9MYmh3eXlXKzZwcEhQWFpWVXdvNUlvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFczRKTzc3bkdrWVgyUGFESjRYNjBOdThuUHpyZWltZVFSUVEvOFEwdm5FY29mbGwwRktMYwprYVEvWTdreEJ5cW81N3dZdkF4NHhrYTJoemk5ZzZ1WjdBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=Other utilities installed are:
- kubectl
- crictl
- k3s-killall.sh
- k3s-uninstall.sh
Install K3s on Worker Nodes
To install on worker nodes we should pass K3S_URL along with K3S_TOKEN or K3S_CLUSTER_SECRET environment variables.
The K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on first node.
$ sudo cat /var/lib/rancher/k3s/server/node-token
K1037242285167f68f26c05e9221751153591384faab8bb6318c3be56a1ff6286e9::server:8d2fb5bc5ca6eb587de0e5920203309bSo to install Kubernetes on worker nodes I’ll run:
k3s_url="https://k3s-master:6443"
k3s_token="K1037242285167f68f26c05e9221751153591384faab8bb6318c3be56a1ff6286e9::server:8d2fb5bc5ca6eb587de0e5920203309b"
curl -sfL https://get.k3s.io | K3S_URL=${k3s_url} K3S_TOKEN=${k3s_token} sh -Installation output:
[INFO] Finding release for channel stable
[INFO] Using v1.27.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3sLogin to one of the master node and check cluster status.
$ sudo kubectl config get-clusters
NAME
default
$ sudo kubectl cluster-info
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-master Ready master 14m v1.27.4+k3s1
k3s-node01 Ready worker 3m11s v1.27.4+k3s1
k3s-node02 Ready worker 3m58s v1.27.4+k3s1
$ sudo kubectl get namespaces
NAME STATUS AGE
default Active 16m
kube-node-lease Active 16m
kube-public Active 16m
kube-system Active 16m
$ sudo kubectl get endpoints -n kube-system
NAME ENDPOINTS AGE
kube-dns 10.42.0.2:53,10.42.0.2:53,10.42.0.2:9153 14m
traefik 10.42.0.5:80,10.42.0.5:443 14m
$ sudo kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-b7464766c-q9frk 1/1 Running 0 15m
helm-install-traefik-8dhpk 0/1 Completed 0 15m
svclb-traefik-9c2j8 2/2 Running 0 4m49s
svclb-traefik-bf9zd 2/2 Running 0 4m2s
svclb-traefik-v2fpx 2/2 Running 0 14m
traefik-5c79b789c5-k589d 1/1 Running 0 14mUse crictl command to see running containers.
# Master
$ sudo crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
acfafb50852d3 18471c10e6e4b 16 minutes ago Running traefik 0 bf8534452389f
fee5ac7e88f2e 4a065d8dfa588 16 minutes ago Running lb-port-443 0 e7068ff7ab2f2
bbab5b07e5efb 4a065d8dfa588 16 minutes ago Running lb-port-80 0 e7068ff7ab2f2
65c5d1333ea04 2ee68ed074c6e 16 minutes ago Running coredns 0 435c51f4716fc
# Workers
$ sudo crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
7ad5c83d6466f 4a065d8dfa588 6 minutes ago Running lb-port-443 0 bf8d9fe57c3f3
c1380eabc0b33 4a065d8dfa588 6 minutes ago Running lb-port-80 0 bf8d9fe57c3f3Uninstall K3s
The k3s installer script will install k3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh. To uninstall K3s, run the command:
$ k3s-uninstall.shCheck K3s documentation for advanced configurations.
Other deployment options for Kubernetes available on our site are:
- Install and Configure Traefik Ingress Controller on Kubernetes Cluster
- Deploy Nginx Ingress Controller on Kubernetes using Helm Chart
- Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray
- How To run Local Kubernetes clusters in Docker
- Deploy Lightweight Kubernetes with MicroK8s and Snap
- Setup Kubernetes / OpenShift Dynamic Persistent Volume Provisioning with GlusterFS and Heketi
