Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Architecture and Concepts

Cluster Operators and Machine Config

See how OpenShift manages itself with Cluster Operators, the Cluster Version Operator and the Machine Config Operator, and how to read their status with oc.

The big idea of OpenShift 4 is that the platform manages itself with the same mechanisms it gives you. The console, the router, the registry, authentication and even the operating system configuration of the nodes are managed by programs called Operators that run inside the cluster. This lesson explains that design and shows how to check whether the cluster is healthy.

What an Operator is

A Kubernetes Operator is a controller that knows how to run one piece of software. You describe what you want in a custom resource, and the Operator creates and repairs the Pods, Services, ConfigMaps and certificates needed. If you change the resource, the Operator rolls out the change. If something drifts or fails, it fixes it.

OpenShift has two families:

  • Cluster Operators are part of the platform itself (about thirty of them). Examples: authentication, console, dns, etcd, image-registry, ingress, kube-apiserver, machine-config, monitoring, network, storage.
  • Add-on Operators are optional software you install from OperatorHub: databases, messaging, service mesh, GitOps. Security and Operators in a later chapter covers installing them.

The Cluster Version Operator (CVO)

The Cluster Version Operator sits above the rest. It reads a release image, a signed list of the exact versions of every Cluster Operator for one OpenShift release, and makes sure each Cluster Operator reaches the matching version. When you upgrade, you tell the CVO a new target version and it orchestrates the entire update in the correct order.

Terminal
oc get clusterversion
oc get clusteroperators
oc adm upgrade

oc get clusterversion shows the current version and whether an update is progressing. oc get clusteroperators (short: oc get co) lists every Cluster Operator with four key columns.

Column Meaning
AVAILABLE The operator's component works
PROGRESSING It is changing something right now
DEGRADED Something is wrong that needs attention
SINCE When the status last changed

A healthy cluster has every operator Available=True, Progressing=False, Degraded=False. During an upgrade you will see Progressing=True on operators that are being updated; that is normal. A Degraded=True for a long time is not. Use oc describe co/<name> to read its conditions and messages.

flowchart TB
  R["Release image<br/>(list of versions)"] --> CVO["Cluster Version Operator"]
  CVO --> A["authentication"]
  CVO --> B["ingress"]
  CVO --> C["machine-config"]
  CVO --> D["monitoring"]
  CVO --> E["...about 30 more"]

Figure: the CVO drives every Cluster Operator to the version in the release image.

The Machine Config Operator (MCO)

Nodes run RHCOS, an image-based OS you do not hand-edit. So how do you change an OS setting such as a kernel argument, a systemd unit or an extra file in /etc? With a MachineConfig object. The Machine Config Operator renders all MachineConfigs that apply to a MachineConfigPool (a group such as master or worker) into one configuration, applies it to each node and reboots the nodes one at a time, keeping the pool available.

YAML
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-motd
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
        - path: /etc/motd
          mode: 0644
          overwrite: true
          contents:
            source: data:,Managed%20by%20MCO%0A

Applying this (read-only example; changing node config is an administrator task) makes the MCO add the file to every worker and reboot them in turn. Check progress with:

Terminal
oc get machineconfigpool
oc get nodes

A pool shows UPDATED, UPDATING and DEGRADED flags. Because configuration changes reboot nodes, make them deliberately and test on a lab cluster first.

Machines and MachineSets

On supported clouds and virtualization, OpenShift can also create the servers. A Machine is a request for one server, a MachineSet keeps N identical Machines (like a ReplicaSet for nodes), and a Cluster Autoscaler can change the MachineSet size. Administrators scale worker capacity with:

Terminal
oc get machinesets -n openshift-machine-api
oc scale machineset <name> -n openshift-machine-api --replicas=5

On bare metal, machine management depends on how the cluster was installed.

Operators are controllers: a model

The standard Operator pattern is custom resource in, running software out. The stand-in below models a tiny operator for a database CR so you see the loop. It is not a real Operator framework.

Python Stand-in: a tiny operator reconciling a custom resource

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading operator health

The stand-in below parses an invented oc get co-style table and reports what a person would look at first. Real output has the same columns, but these rows are made up.

Python Stand-in: summarize cluster operator health

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why this matters to you

Because Operators own configuration, you should not edit their managed objects directly. If you hand-edit a Deployment in openshift-ingress, the Ingress Operator may undo your change. You change the Operator's configuration resource (for example IngressController) or a documented cluster-level object instead. The same rule applies to MachineConfigs: never edit files on nodes by hand and expect them to stay.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the Cluster Version Operator do?

    The CVO orchestrates platform updates by driving each Cluster Operator to the version in the release image.
  2. 2. A Cluster Operator shows DEGRADED=True for a long time. What should you do?

    Degraded means something needs attention. The operator's conditions and messages explain what.
  3. 3. How do you change a kernel argument on every worker node?

    MachineConfig objects are the supported way to change RHCOS; the Machine Config Operator applies them and reboots nodes in turn.
  4. 4. What is a MachineSet?

    A MachineSet is to Machines what a ReplicaSet is to Pods.

Summary

  • Cluster Operators run the platform; add-on Operators from OperatorHub run optional software.
  • The CVO upgrades all Cluster Operators according to a release image.
  • oc get co shows AVAILABLE, PROGRESSING and DEGRADED; investigate with oc describe co/<name>.
  • The Machine Config Operator applies MachineConfigs to RHCOS nodes and reboots them gradually.
  • Change the Operator's configuration resource, never its managed objects by hand.

Next: API Objects and Inspecting Them with oc