After setting up your Kubernetes cluster in the cloud or on an on-premise environment, there will be a need to check what is going on inside your warehouse in an easy and flexible fashion. One of the best ways to do that is to investigate logs whenever you need to fix or know what happened at a particular time. While it is possible to log into your cluster and check the logs of your pods or your hosts, it suddenly becomes tedious to check each pod’s logs individually especially if you have many pods in your k8s. To make it easier for you to check the status of your cluster on one platform, we are going to deploy Elasticsearch and Kibana on an external server then ship logs from your cluster to Elasticsearch using Elastic’s beats (Filebeat, Metricbeat etc). If you already have an ELK Stack already running, then the better.
The diagram below shows the architecture we shall accomplish in this guide. It is essentially a 3 node Kubernetes cluster and one Elasticsearch and Kibana server which will be receiving logs from the cluster via Filebeat and Metricbeat log collectors.
First, we shall need an Elasticsearch server with with Kibana installed as well. You can leave out Logstash but incase you need further filtering of your logs, then you can have it installed. Follow the guides below to install Elasticsearch and Kibana:
- How To Install ElasticSearch 7.x on CentOS 7
- How To Install Elasticsearch 7 on Debian
- How To Install Elasticsearch on Ubuntu
On your Elasticsearch host, make sure it can be reached from outside. You can edit the following parts in your config
$ sudo vim /etc/elasticsearch/elasticsearch.yml
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#Then allow the port on your firewall
sudo firewall-cmd --permanent --add-port=9200/tcp
sudo firewall-cmd --reload
Secondly, you must have a Kubernetes cluster since it is the place we are concerned about fetching logs from. We have a couple of guides that will help you set one up in case you need to bootstrap one quickly. They are shared below:
Install Kubernetes Cluster on Ubuntu with kubeadm
Install Kubernetes Cluster on CentOS 7 with kubeadm
Easily Setup Kubernetes Cluster on AWS with EKS
Deploy Kubernetes Cluster with Ansible & Kubespray
Install Production Kubernetes Cluster with Rancher RKE
After you are ready, we can proceed to install Filebeat and Metricbeat pods in our cluster to start collecting and sending the logs to ELK. Make sure you are able to run kubectl command in your Kubernetes cluster.
Step 1: Download Sample Filebeat and Metricbeat files
Log into your Kubernetes master node and run the command below to fetch Filebeat and Metricbeat yaml files provided by Elastic.
cd ~
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/filebeat-kubernetes.yaml
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/metricbeat-kubernetes.yamlStep 2: Edit the files to befit your environment
In both files, we only need to change a few things. Under the ConfigMap, you will find elasticseach output as shown below. Change the ip (192.168.10.123) and port (9200) to that of your Elasticsearch server.
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:192.168.10.123}:${ELASTICSEARCH_PORT:9200}']
#username: ${ELASTICSEARCH_USERNAME}
#password: ${ELASTICSEARCH_PASSWORD}
Under DaemonSet within the same file, you will find the following configuration. Note that we are showing the areas to change. Edit the IP (192.168.10.123) and port (9200) to match that of your Elastcsearch server as well. If you have configured username and password for your Elasticsearch, you are free to add them on the commented sections shown.
env:
- name: ELASTICSEARCH_HOST
value: "192.168.10.123"
- name: ELASTICSEARCH_PORT
value: "9200"
#- name: ELASTICSEARCH_USERNAME
# value: elastic
#- name: ELASTICSEARCH_PASSWORD
# value: changeme
- name: ELASTIC_CLOUD_IDIt should be noted that if you would wish to deploy your Filebeat and Metricbeat resources on another namespace, simply edit all instances of “kube-system” to the one of your choice.
Under Deployment, you can change the version of Filebeat and Metricbeat to be deployed by editing the image (docker.elastic.co/beats/metricbeat:7.9.0) seen on the snippet below. I am going to use version 7.9.0.
###For Metricbeat####
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:7.9.0Do the same for the filebeat yml file if you would wish to change its version as well.
###For Filebeat####
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/filebeat:7.9.0
Important thing to note
If you would wish to deploy the beats on the master node, we will have to add tolerations. An example for metricbeat is shown below. That is not the whole DaemonSet config. Just the part we are interested in. You can leave the other parts intact. Add the toleration as shown on the configuration below under spec. The same can apply on filebeat configuration depending on your needs.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
selector:
matchLabels:
k8s-app: metricbeat
template:
metadata:
labels:
k8s-app: metricbeat
spec:
###PART TO EDIT###
# This toleration is to have the daemonset runnable on master nodes
# Remove it if your masters can't run pods
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
####END OF EDIT###
serviceAccountName: metricbeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:7.9.0
args: [
"-c", "/etc/metricbeat.yml",
"-e",Step 3: Deploying to Kubernetes
After we have done all of our edits and our Elasticsearch is well reachable from your Kubernetes cluster, It is time to deploy our beats. Login to your master node and run the commands below:
kubectl apply -f metricbeat-kubernetes.yaml
kubectl apply -f filebeat-kubernetes.yamlConfirm that the pods are deployed and running successfully after some time.
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-c9784d67d-k85hf 1/1 Running 5 11d
calico-node-brjnk 1/1 Running 7 10d
calico-node-nx869 1/1 Running 1 10d
calico-node-whlzf 1/1 Running 6 11d
coredns-f9fd979d6-6vztd 1/1 Running 5 11d
coredns-f9fd979d6-8gz4l 1/1 Running 5 11d
etcd-kmaster.diab.mfs.co.ke 1/1 Running 5 11d
filebeat-hlzhc 1/1 Running 7 7d23h <==
filebeat-mcs67 1/1 Running 1 7d23h <==
kube-apiserver-kmaster.diab.mfs.co.ke 1/1 Running 5 11d
kube-controller-manager-kmaster.diab.mfs.co.ke 1/1 Running 5 11d
kube-proxy-nlrbv 1/1 Running 5 11d
kube-proxy-zdcbg 1/1 Running 1 10d
kube-proxy-zvf6c 1/1 Running 7 10d
kube-scheduler-kmaster.diab.mfs.co.ke 1/1 Running 5 11d
metricbeat-5fw98 1/1 Running 7 8d <==
metricbeat-5zw9b 1/1 Running 0 8d <==
metricbeat-jbppx 1/1 Running 1 8d <==Step 4: Create Index on Kibana
Once our Pods begin running, they will immediately send an index pattern to Elasticsearch together with the logs. Login to your Kibana and Click “Stack Management” > “Index Management” and you should be able to see your indexes.
Click on “Index Management“
And there are our indexes.
To create an index pattern, click on “Index Patterns” then hit “Create Index Pattern“.
On the next page, type in the name of the index pattern that matches either filebeat or metricbeat and they should show up as already matched.
Create the pattern then click “Next Step“
Choose “@timestamp” on the drop-down then “Create Index Pattern“
Step 5: Discover your Data
After the index pattern has been created, click on “Discover”,
Then choose the index pattern we created.
Conclusion
Now we have lightweight beats that fetch logs and metrics from your Kubernetes cluster and ship them to an external Elasticsearch for indexing and flexible searching.
