So you have your OpenShift Cluster and you’re now on standard day 2 operations?. Checking the Logs of your Cluster Nodes is a normal operation when troubleshooting Cluster issues, e.g failed services, network & Storage issues e.t.c. In this guide, we’ll explore how one can display OpenShift Node Logs using the command line tools.
As from OpenShift Container Platform 4.0, OpenShift nodes are based on Red Hat Enterprise Linux CoreOS (RHCOS) and OKD versions uses Fedora CoreOS (FCOS). These operating systems run very few local services that would require direct access to a node to inspect their status.
Display Logs of OpenShift Nodes
Most of the system services in RHCOS / FCOS run as containers with the exceptions of the CRI-O container engine and the Kubelet, which are Systemd units.
sh-4.4# systemctl status crio
● crio.service - Open Container Initiative Daemon
Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/crio.service.d
└─10-default-env.conf
Active: active (running) since Sat 2020-03-28 10:59:50 UTC; 1h 7min ago
Docs: https://github.com/cri-o/cri-o
Main PID: 2937 (crio)
Tasks: 42
Memory: 322.4M
CPU: 5min 28.410s
CGroup: /system.slice/crio.service
└─2937 /usr/bin/crio --enable-metrics=true --metrics-port=9537
.....
sh-4.4# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-default-env.conf
Active: active (running) since Sat 2020-03-28 11:00:11 UTC; 1h 7min ago
Process: 3028 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
Process: 3026 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Main PID: 3030 (hyperkube)
Tasks: 56 (limit: 26213)
Memory: 241.6M
CPU: 5min 7.547s
CGroup: /system.slice/kubelet.service
└─3030 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeco
.....
The oc adm node-logs command can be used to view these logs without requiring direct ssh to the cluster nodes. The syntax is:
oc adm node-logs [-l LABELS] [NODE...] [flags]The command above will display all journal logs of a node. Example:
$ oc adm node-logs node01.ocp.neveropen.techTo return log entries from the specified unit(s), use the -u option:
$ oc adm node-logs <node-name> -u crioShow logs available in a node in /var/log:
$ oc adm node-logs node01.ocp.geeksforgeeks.org --path=/
audit/
btmp
chrony/
containers/
crio/
es-containers.log.pos
fluentd/
glusterfs/
journal/
journal_pos.json
lastlog
openvswitch/
pods/
private/
samba/
sssd/
vmware-network.1.log
vmware-network.2.log
vmware-network.log
vmware-vgauthsvc.log.0
vmware-vmsvc.1.log
vmware-vmsvc.log
wtmp
Display log file from a path in a node:
oc adm node-logs node01.ocp.geeksforgeeks.org --path=/audit/audit.logShow kubelet logs from all masters / workers:
oc adm node-logs --role master -u kubelet
oc adm node-logs --role worker -u kubeletSee what logs are available in masters/workers/infra nodes in /var/log:
oc adm node-logs --role master --path=/
oc adm node-logs --role worker --path=/
oc adm node-logs --role infra --path=/OpenShift Courses:
More articles on OpenShift Container Platform:
