Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Administration

Cluster Updates, MachineSets and Machine Config in OpenShift

Update an OpenShift cluster safely: channels, the Cluster Version Operator, update order, Machine Config Operator, MachineConfigPools, MachineSets, pause and rollback limits.

OpenShift 4 treats the operating system, the platform components and the cluster configuration as one managed product. An update is a single, orchestrated operation: new platform operators roll out, then the nodes' operating system (Red Hat Enterprise Linux CoreOS, RHCOS) is updated and nodes reboot one by one. This lesson explains who does what and how to run updates with little risk.

The product is not available here: the commands are read-only examples, and the Python stand-ins model version planning and rolling node updates.

Who updates what

  • The Cluster Version Operator (CVO) reads the update graph from a channel and applies a release image, which contains the versions of every cluster operator.
  • Each cluster operator updates its component, then reports Available, Progressing and Degraded (operators).
  • The Machine Config Operator (MCO) updates node operating systems and kubelet configuration, one pool at a time.
Terminal
oc get clusterversion
oc adm upgrade                     # shows current version, channel and available updates
oc adm upgrade channel stable-4.18
oc adm upgrade --to=4.18.12        # start an update (or --to-latest)
oc get clusteroperators            # watch operators go Progressing then back
oc get mcp                         # MachineConfigPools: UPDATED / UPDATING / DEGRADED

(The version numbers above are placeholders. Take the real ones from oc adm upgrade.)

Channels

Channel Purpose
candidate-4.y Early builds, no support commitment for production
fast-4.y Updates available as soon as they pass checks
stable-4.y Updates delayed until proven by more clusters
eus-4.y Extended update support for even-numbered minor releases

Pick stable for production. Updating between minor versions (4.17 to 4.18) means switching channel, and the graph only offers supported paths. You generally cannot skip a minor version (4.16 straight to 4.18); updates go through each minor, although EUS channels allow a controlled path between two even releases.

Python Stand-in: find an update path

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The graph in this exercise is invented to show the idea: the real one is served by Red Hat and lists many more edges.

Before you update

  1. Read the release notes and the update documentation for deprecated or removed APIs (oc get apirequestcounts shows which API versions your clients still use).
  2. Make sure oc get clusteroperators is healthy and that nodes are all Ready.
  3. Back up etcd (/usr/local/bin/cluster-backup.sh run from a control-plane node through oc debug node/...).
  4. Check that your add-on operators support the target version (OLM warns in the console when an update is blocked through upgradeable=False).
  5. Make sure workloads tolerate a node reboot: at least two replicas, PodDisruptionBudgets, readiness probes.

PodDisruptionBudgets

During node drains, a PDB tells the cluster how many pods of an app may be unavailable:

YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shop-api
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: shop-api

A PDB that allows zero disruptions (a single replica with minAvailable: 1) blocks the drain and stalls the update. Check with oc get pdb -A.

Machine Config Operator

Node configuration (kernel arguments, files in /etc, systemd units, chrony time settings) is declared as MachineConfig objects and applied to nodes through MachineConfigPools such as master and worker:

YAML
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-motd
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
        - path: /etc/motd
          mode: 0644
          overwrite: true
          contents:
            source: data:,Managed%20by%20the%20Machine%20Config%20Operator%0A

The MCO merges all MachineConfigs for the pool into one rendered config. When it changes, the MCO cordons, drains, updates and reboots one node at a time (controlled by maxUnavailable, default 1), then moves on. Because most changes reboot nodes, batch them. You can pause a pool to hold changes: oc patch mcp/worker --type merge -p '{"spec":{"paused":true}}', but do not leave it paused for long, as certificates need rotation.

Python Stand-in: rolling node update

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the run time with max_unavailable = 1. A higher value is faster but removes more capacity at once; only raise it if the remaining nodes can carry the load.

MachineSets and node scaling

On installer-provisioned platforms, MachineSets manage worker machines the way a ReplicaSet manages pods. Changing replicas adds or removes cloud instances:

Terminal
oc get machinesets -n openshift-machine-api
oc scale machineset worker-us-east-1a --replicas=4 -n openshift-machine-api
oc get machines -n openshift-machine-api
oc get nodes

A MachineSet defines the instance type, image and subnet; create one per availability zone, or per hardware shape (for example GPU nodes). The Cluster Autoscaler (autoscaling) adjusts these replicas through MachineAutoscaler resources. On bare metal or user-provisioned clusters you add nodes yourself and approve their certificate requests with oc adm certificate approve.

During and after the update

Terminal
oc get clusterversion -w
oc adm upgrade                              # progress summary
oc get nodes                                # nodes cycle through SchedulingDisabled
oc get mcp                                  # UPDATING True while nodes reboot

After completion verify: oc get clusteroperators all healthy, nodes Ready, representative apps working, and alerts quiet. Rollback is not supported for a whole cluster in general; the safety net is etcd backup and, for important environments, rehearsing the update on a non-production cluster first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component applies a new release image across the cluster?

    The CVO reads the update graph and rolls out the release payload.
  2. 2. What does the Machine Config Operator do during node configuration changes?

    Pools update with maxUnavailable limiting concurrent nodes, default one.
  3. 3. Why can a single-replica app with a PDB of minAvailable 1 stall an update?

    A PDB allowing zero disruptions blocks eviction, so the node never drains.
  4. 4. Which channel is typically used for production clusters?

    Stable delays updates until they have proven themselves on many clusters.

Summary

  • The CVO, cluster operators and the MCO update the platform and node OS together.
  • Choose a channel (stable for production), follow supported paths, read the notes first.
  • Prepare: etcd backup, healthy operators, PDBs, multiple replicas.
  • MachineConfigs change nodes through pools, one node at a time with reboots.
  • MachineSets scale worker machines; rollback is not a standard option.

Next: Project: deploy a three-tier app