After a successful installation and configuration of OpenShift Container Platform, the updates are providedover-the-air by OpenShift Update Service (OSUS). The operator responsible for checking valid updates available for your cluster with the OpenShift Update Service is called Cluster Version Operator (CVO). When you request an update, the CVO uses the release image for that update to upgrade your cluster. All the release artifacts are stored as container images in the Quay registry.
It is important to note that the OpenShift Update Service displays all valid updates for your Cluster version. It is highly recommended that you do not force an update to a version that the OpenShift Update Service does not display. This is because a suitability check is performed to guarantee functional cluster after the upgrade. During the upgrade process, the Machine Config Operator (MCO) applies the new configuration to your cluster machines.
Before you start a minot upgrade to your OpenShift Cluster, check the current cluster version using oc command line tool if configured or from a web console. You should have the cluster admin rolebinding to use these functions.
We have the following OpenShift / OKD installation guides on our website:
- How To Deploy OpenShift Container Platform 4.x on KVM
- How To Install OKD OpenShift 4.x Cluster on OpenStack
- Setup Local OpenShift 4.x Cluster with CodeReady Containers
1) Confirm current OpenShift Cluster version
Check the current version and ensure your cluster is available:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.5 True False 24d Cluster version is 4.8.5The current version of OpenShift Container Platform installed can also be checked from the web console – Administration → Cluster Settings > Details
Also check available Cluster nodes and their current status. Ensure they are all in Ready State before you can initiate an upgrade.
$ oc get nodes
NAME STATUS ROLES AGE VERSION
master01.ocp4.techwizpro.com Ready master 24d v1.21.1+9807387
master02.ocp4.techwizpro.com Ready master 24d v1.21.1+9807387
master03.ocp4.techwizpro.com Ready master 24d v1.21.1+9807387
worker01.ocp4.techwizpro.com Ready worker 24d v1.21.1+9807387
worker02.ocp4.techwizpro.com Ready worker 24d v1.21.1+9807387
worker03.ocp4.techwizpro.com Ready worker 24d v1.21.1+9807387
2) Backup Etcd database data
Access one of the control plane nodes(master node) using oc debug command to start a debug session:
$ oc debug node/<node_name>Here is an example with expected output:
$ oc debug node/master01.ocp4.example.com
Starting pod/master01ocp4examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.100.11
If you don't see a command prompt, try pressing enter.
sh-4.4#Change your root directory to the host:
sh-4.4# chroot /host
Then initiate backup of etcd data using provided script namedcluster-backup.sh:
sh-4.4# which cluster-backup.sh
/usr/local/bin/cluster-backup.shThe cluster-backup.sh script is part of etcd Cluster Operator and it is just a wrapper around the etcdctl snapshot save command. Execute the script while passing the backups directory:
sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backupHere is the output as captured from my backup process
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-19
found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-9
found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
3f8cc62fb9dd794113201bfabd8af4be0fdaa523987051cdb358438ad4e8aca6
etcdctl version: 3.4.14
API version: 3.4
{"level":"info","ts":1631392412.4503953,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db.part"}
{"level":"info","ts":"2021-09-11T20:33:32.461Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1631392412.4615548,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://157.90.142.231:2379"}
{"level":"info","ts":"2021-09-11T20:33:33.712Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1631392413.9274824,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://157.90.142.231:2379","size":"102 MB","took":1.477013816}
{"level":"info","ts":1631392413.9344463,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db"}
Snapshot saved at /home/core/assets/backup/snapshot_2021-09-11_203329.db
{"hash":3708394880,"revision":12317584,"totalKey":7946,"totalSize":102191104}
snapshot db and kube resources are successfully saved to /home/core/assets/backup
Check if the backup files are available in our backups directory:
sh-4.4# ls -lh /home/core/assets/backup/
total 98M
-rw-------. 1 root root 98M Sep 11 20:33 snapshot_2021-09-11_203329.db
-rw-------. 1 root root 92K Sep 11 20:33 static_kuberesources_2021-09-11_203329.tar.gzThe files as seen are:
snapshot_<datetimestamp>.db: The etcd snapshot file.static_kuberesources_<datetimestamp>.tar.gz: File that contains the resources for the static pods. When etcd encryption is enabled, the encryption keys for the etcd snapshot will be contained in this file.
You can copy the backup files to a separate system or location outside the server for better security if the node becomes unavailable during upgrade.
3) Changing Updates Channel (Optional)
The OpenShift Container Platform offers the following upgrade channels:
- candidate
- fast
- stable
Review the current update channel information and confirm that your channel is set to stable-4.8:
$ oc get clusterversion -o json|jq ".items[0].spec"
{
"channel": "fast-4.8",
"clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
}You can decide to change an upgrade channel before the actual upgrade of the cluster.
From Command Line Interface
Switch Update channel from CLI using patch:
oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "<channel>”}]'
# Example
$ oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "stable-4.8"}]'
clusterversion.config.openshift.io/version patched
$ oc get clusterversion -o json|jq ".items[0].spec"
{
"channel": "stable-4.8",
"clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
}From Web Console
NOTE:For production clusters, you must subscribe to a stable-* or fast-* channel. Your cluster is fully supported by Red Hat subscription if you change from stable to fast channel.
In my example below I’ve set the channel to fast-4.8.
4) Perform Minor Upgrade on OpenShift / OKD Cluster
You can choose to perform a cluster upgrade from:
- Bastion Server / Workstation oc command line
- From OpenShift web console
Upgrade your OpenShift Container Platform from CLI
Check available upgrades
$ oc adm upgrade
Cluster version is 4.8.5
Updates:
VERSION IMAGE
4.8.9 quay.io/openshift-release-dev/ocp-release@sha256:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd
4.8.10 quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364dbAs seen we have two minor upgrades that can be performed:
- To version 4.8.9
- To version 4.8.10
The easiest way to upgrade is to the latest version:
$ oc adm upgrade --to-latest=true
Updating to latest version 4.8.10To update to a specific version:
$ oc adm upgrade --to=<version>
#e.g 4.8.9, I'll run:
$ oc adm upgrade --to=4.8.9 You can easily review Cluster Version Operator status with the following command:
$ oc get clusterversion -o json|jq ".items[0].spec"
{
"channel": "stable-4.8",
"clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585",
"desiredUpdate": {
"force": false,
"image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
"version": "4.8.10"
}
}The oc adm upgrade command will give progress update with the steps:
$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.8.10: 69 of 678 done (10% complete)
Updates:
VERSION IMAGE
4.8.9 quay.io/openshift-release-dev/ocp-release@sha256:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd
4.8.10 quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364dbUpgrade OpenShift Container Platform from UI
Administration → Cluster Settings→ Details→ Select channel→ Select a version to update to, and click Save. The Input channel Update status changes to Update to <product-version> in progress.
All cluster operators will be upgraded one after the other until all are in the minor version selected during upgrade:
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.8.5 True False False 119m
baremetal 4.8.5 True False False 24d
cloud-credential 4.8.5 True False False 24d
cluster-autoscaler 4.8.5 True False False 24d
config-operator 4.8.5 True False False 24d
console 4.8.5 True False False 36h
csi-snapshot-controller 4.8.5 True False False 24d
dns 4.8.5 True False False 24d
etcd 4.8.10 True False False 24d
image-registry 4.8.5 True False False 24d
ingress 4.8.5 True False False 24d
insights 4.8.5 True False False 24d
kube-apiserver 4.8.5 True False False 24d
kube-controller-manager 4.8.5 True False False 24d
kube-scheduler 4.8.5 True False False 24d
kube-storage-version-migrator 4.8.5 True False False 4d16h
machine-api 4.8.5 True False False 24d
machine-approver 4.8.5 True False False 24d
machine-config 4.8.5 True False False 24d
marketplace 4.8.5 True False False 24d
monitoring 4.8.5 True False False <invalid>
network 4.8.5 True False False 24d
node-tuning 4.8.5 True False False 24d
openshift-apiserver 4.8.5 True False False 32h
openshift-controller-manager 4.8.5 True False False 23d
openshift-samples 4.8.5 True False False 24d
operator-lifecycle-manager 4.8.5 True False False 24d
operator-lifecycle-manager-catalog 4.8.5 True False False 24d
operator-lifecycle-manager-packageserver 4.8.5 True False False 7d11h
service-ca 4.8.5 True False False 24d
storage 4.8.5 True False False 24d5) Validate OpenShift CLuster Upgrade
Wait for the upgrade process to complete then confirm that the cluster version has updated to the new version:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.10 True False 37h Cluster version is 4.8.10Checking cluster version from the web console
To obtain more detailed information about the cluster status run the command:
$ oc describe clusterversionIf you try running the command oc adm upgrade immediately after upgrade to the latest release you should get a message similar to below:
$ oc adm upgrade
Cluster version is 4.8.10
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.Conclusion
In this short guide we’ve shown how one can easily perform minor upgrade of OpenShift container cluster version. The process can be initiated from a web console or from the command line, it all depends on your preference. In our articles to follow we’ll cover steps required to perform Major versions upgrade in anOpenShift container cluster.
More guides on OpenShift / OKD container Platform:
