Skip to content
Tutorials Center
My learning

Topics

Lesson 64 of 78 ยท Operations and Observability

Cluster Operations and Upgrades

Upgrade a Kubernetes cluster safely: version skew rules, upgrade order, node pools, etcd snapshots and restores, certificate expiry and managed vs self-managed.

A Kubernetes cluster is not something you install once. A new minor version appears roughly every four months, and each version is supported with security patches for about a year. A cluster that is never upgraded ends up out of support, and the longer you wait, the more risky and painful the jump becomes. Upgrading is a routine job, and the way to make it boring is to understand a few rules: the version skew policy, the upgrade order, and what to back up before you start.

Everything here is read-only: the commands are real, but this site has no cluster to run them on. The Python stand-ins check version skew, plan a multi-step upgrade and watch certificate expiry, so you can practice the rules without touching a real cluster. Version numbers and skew limits shown are for the current policy; always check the official skew policy for your exact versions before planning.

Versions and what "supported" means

Kubernetes versions look like 1.34.2: major.minor.patch. Patch releases (1.34.1 to 1.34.2) fix bugs and security problems and are safe and quick. Minor releases (1.33 to 1.34) add features, deprecate and remove APIs (API versions) and are where you must read the release notes. The project maintains roughly the three most recent minor versions.

Version vocabulary

Patch Bug and security fixes. Low risk.
Minor New features and API removals. Read the notes.
Support window About a year per minor version.

Plan one minor upgrade a few times a year, not a big jump every three years.

1 / 2

The version skew policy

During an upgrade the components of a cluster briefly run different versions, so Kubernetes defines how far apart they may be. The API server is the reference point. In the current policy:

  • kube-apiserver instances in a highly available control plane may differ by at most one minor version.
  • kubelet must not be newer than the API server, and may be up to three minor versions older.
  • kube-proxy follows the same limits as the kubelet.
  • kube-controller-manager, kube-scheduler and cloud-controller-manager must not be newer than the API server and may be one minor version older.
  • kubectl may be one minor version older or newer than the API server.

The key idea: the API server is upgraded first and everything else follows. Nothing may be newer than the API server (except kubectl, within one version).

Python A version skew checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first cluster is fine: its kubelet is two minors behind (allowed) and kubectl is one ahead (allowed). In the second cluster the kubelet is four minors behind, the scheduler is newer than the API server and kubectl is three minors behind. A real upgrade tool checks these for you, but knowing the rules lets you see why an upgrade plan is ordered the way it is.

Upgrade order

You cannot skip minor versions: going from 1.32 to 1.34 means 1.32 to 1.33 to 1.34. At each minor step the order is:

  1. Read the release notes and the deprecated API list; run a check for objects that use removed APIs. Back up etcd.
  2. Upgrade the control plane (API server first, then controller manager, scheduler).
  3. Upgrade add-ons: CNI plugin, CoreDNS, kube-proxy, metrics-server, ingress controller, as the new version requires.
  4. Upgrade the nodes, a few at a time.

For nodes, the standard rolling procedure drains a node (evicts its pods, respecting PodDisruptionBudgets), upgrades it and puts it back:

Terminal
kubectl cordon worker-1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
# ... upgrade kubelet and the runtime on the node, restart kubelet ...
kubectl uncordon worker-1

On a self-managed cluster built with kubeadm, the control plane steps look like this:

Terminal
kubeadm upgrade plan
sudo kubeadm upgrade apply v1.34.2     # first control plane node
sudo kubeadm upgrade node              # other control plane nodes and workers

Each node also needs the new kubelet package installed and the service restarted. On managed services, the cloud provider does the control plane upgrade with one command or click, and node pools are upgraded either in place (rolling) or, in the safer blue-green style, by creating a new pool with the new version, moving workloads over and deleting the old pool.

The next stand-in plans a multi-step upgrade for you: it walks one minor at a time, upgrades the control plane first and then the nodes, and checks the skew after every step.

Python Plan a multi-step upgrade

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plan has six steps, alternating control plane and nodes. The last lines show the lag allowance: nodes may stay three minors behind the API server during a long upgrade campaign, but the limit means you cannot postpone the node work forever.

Backing up etcd

etcd holds all cluster state: every Deployment, Secret and ConfigMap. If the control plane is lost and you have no etcd backup, you lose the cluster's configuration (your applications' data on volumes is separate). Take a snapshot before every upgrade and on a schedule:

Terminal
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-2026-11-06.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdutl snapshot status /backup/etcd-2026-11-06.db --write-out=table

To restore, stop the API server, restore the snapshot into a new data directory with etcdutl snapshot restore /backup/etcd-2026-11-06.db --data-dir /var/lib/etcd-restored, point the etcd static pod at that directory, and start again. Practice a restore on a test cluster; a backup you have never restored is only a hope. Store snapshots off the machine and encrypt them, since they contain Secrets (secrets). Managed services (EKS, GKE, AKS) hide etcd and back it up for you, but you still need your own backup of application data and manifests, for example with Velero.

Certificates

The control plane components talk to each other over TLS. On kubeadm clusters, the leaf certificates are valid for one year and are renewed automatically when you run kubeadm upgrade, while the cluster CAs last ten years. A cluster that is never upgraded will suddenly stop working a year after install when its certificates expire, a classic outage.

Terminal
sudo kubeadm certs check-expiration
sudo kubeadm certs renew all

After renewing, restart the control plane static pods so they load the new certificates. The kubelet client certificates normally rotate by themselves.

Python Certificate expiry check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The etcd server certificate has 14 days left and the front proxy certificate 54, so both are flagged. In a real cluster, monitoring should alert long before a certificate gets this close.

Managed and self-managed

Who does the work

Managed The provider runs the control plane and etcd.
Self-managed You run everything, including etcd and certificates.

Managed does not mean hands-off. You still upgrade node pools and add-ons.

1 / 2

For most teams, a managed service is the right default: the provider handles control plane high availability, etcd backups and certificates. Self-managed (kubeadm, k3s, Talos and others) gives full control, and is common on-premises, but you own every task in this lesson. Either way the loop is the same: read notes, test in staging, upgrade the control plane, then nodes, then verify.

What is real and what is a stand-in

The commands (kubeadm, etcdctl, etcdutl, kubectl drain) and skew rules are the real ones, but exact flags and limits change between versions, so check the current documentation. The Python code encodes the skew rules and the date arithmetic only; it cannot know about removed APIs or add-on compatibility.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In which order do you upgrade a cluster?

    No component may be newer than the API server, so the control plane goes first and nodes follow.
  2. 2. Your cluster is on 1.31. Can you go straight to 1.34 in one upgrade?

    Control plane upgrades go one minor version at a time, because each release only supports migrating from the previous one.
  3. 3. What should you always do before a control plane upgrade?

    etcd stores all cluster state, so a snapshot allows recovery. A scan for removed API versions prevents manifests from breaking.
  4. 4. Why can a kubeadm cluster that was never upgraded suddenly fail after about a year?

    kubeadm leaf certificates are valid for one year. kubeadm upgrade renews them, so a never-upgraded cluster hits expiry.

Summary

  • Patch upgrades are quick; minor upgrades need release notes, an API-removal check and one step at a time.
  • Skew policy: the API server is the reference; kubelet and kube-proxy may be up to three minors older; scheduler and controller manager up to one older; kubectl within one either way.
  • Upgrade order: back up etcd, control plane, add-ons, then nodes via cordon and drain (honoring PodDisruptionBudgets).
  • Snapshot etcd with etcdctl snapshot save, restore with etcdutl snapshot restore, and practice the restore.
  • kubeadm certificates last a year; upgrades renew them, and monitoring should warn before they expire.
  • Managed services run the control plane for you, but node pools, add-ons and API deprecations remain your job.

Next: Cost, Reliability and Multi-Cluster, on spending less and surviving failures.