Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 78 ยท Operations and Observability

Nodes: Status, Conditions, Leases, Cordon and Drain

Understand Kubernetes nodes: registration, capacity and allocatable, conditions, NotReady causes, lease heartbeats, cordon and drain, and graceful or non-graceful shutdown.

A node is a machine (physical or virtual) that runs Pods. Everything you deploy ends up on one: the scheduler picks a node, and the kubelet on that node starts the containers. When a node misbehaves, Pods on it misbehave, so knowing how Kubernetes sees a node, how it decides a node is sick, and how to take one out of service safely is core operations knowledge.

This lesson covers how nodes register, what the Node object tells you, how heartbeats and conditions work, what happens when a node stops answering, and the maintenance commands cordon and drain. There is no cluster on this site, so every manifest and kubectl command is read-only. The two runnable examples are Python stand-ins that model the controller rules; they are not Kubernetes.

How a node joins

A node is not "created" by an administrator in the usual case. The kubelet registers itself with the API server when it starts (--register-node=true is the default), using the credentials it got when it joined the cluster (Building a Cluster with kubeadm). It sends a Node object with its name, addresses, labels and resources. From then on, the kubelet keeps that object up to date, and the control plane treats it as the source of truth about the machine.

Terminal
kubectl get nodes
kubectl get nodes -o wide
kubectl describe node worker-a
TEXT
NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   41d   v1.36.1
worker-a   Ready    <none>          41d   v1.36.1
worker-b   Ready    <none>          41d   v1.36.1

The name must be unique and, for managed clouds, matches the machine's identity. If the machine is replaced, the old Node object stays until something (a cloud controller, an autoscaler or you) deletes it.

What is inside a Node object

kubectl describe node is the best single view. The parts that matter:

Section What it holds
Labels OS, architecture, hostname, zone, instance type, roles. Used by selectors and affinity (Node Selectors and Affinity).
Taints Rules that repel Pods unless tolerated (Taints and Tolerations).
Addresses InternalIP, ExternalIP, Hostname.
Capacity The machine's total CPU, memory, ephemeral storage and Pod count.
Allocatable What is left for Pods after reserving resources for the OS and Kubernetes itself. The scheduler uses this number.
Conditions Health flags (below).
System Info Kernel, OS image, kubelet version, container runtime.
Non-terminated Pods and Allocated resources What is running there and how much of Allocatable is requested (Resource Requests and Limits).

Capacity vs Allocatable explains a common surprise: a 4 CPU / 16 GiB node might offer only 3.9 CPU and about 14 GiB to Pods, because the kubelet reserves some for system daemons (--kube-reserved, --system-reserved) and keeps a hard eviction threshold free. Always do capacity math with Allocatable.

Useful labels that every node carries:

TEXT
kubernetes.io/hostname=worker-a
kubernetes.io/os=linux
kubernetes.io/arch=amd64
topology.kubernetes.io/region=eu-central
topology.kubernetes.io/zone=eu-central-1a
node.kubernetes.io/instance-type=m6i.large

The topology.kubernetes.io/* labels are set by the cloud provider or by you. Topology spread constraints, volume placement and zone-aware routing all read them (Node Selectors and Affinity, The Scheduler).

Conditions

The kubelet reports a list of conditions, each True, False or Unknown:

Condition Meaning when True
Ready The node is healthy and can accept Pods. (False = unhealthy, Unknown = the control plane has not heard from it.)
MemoryPressure The node is low on memory.
DiskPressure The node is low on disk space or inodes.
PIDPressure Too many processes.
NetworkUnavailable The node's network has not been configured (set by the CNI or cloud controller).

The three pressure conditions come from eviction thresholds in the kubelet. When memory is short, the kubelet sets MemoryPressure=True, the node gets a node.kubernetes.io/memory-pressure:NoSchedule taint (no new Pods) and it evicts Pods, starting with those using more than they requested (Resource Requests and Limits, Disruptions and Priority).

Terminal
kubectl get node worker-a -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'
TEXT
MemoryPressure=False DiskPressure=False PIDPressure=False Ready=True

Why a node goes NotReady

Ready=False means the kubelet itself says it is unhealthy. Common causes, in the order you should check them:

  1. The kubelet is down or crashing. On the node, systemctl status kubelet and journalctl -u kubelet.
  2. The container runtime is down (containerd/CRI-O), so the kubelet cannot run Pods.
  3. Network is not ready because the CNI plugin is missing or crashed (CNI and Pod Networking).
  4. Resource exhaustion: disk full, out of memory, PID limit.
  5. Certificates expired so the kubelet cannot talk to the API server.

Ready=Unknown is different: the kubelet did not say anything, and the control plane lost contact. That is a machine that is off, a network partition, or a kubelet that is hung.

Heartbeats: status updates and leases

How does the control plane know a node is alive? Two mechanisms:

  • Node status updates. The kubelet patches .status of the Node when something changes, and at least every few minutes otherwise (nodeStatusReportFrequency, 5 minutes by default).
  • Leases. Each node has a small Lease object in the kube-node-lease namespace. The kubelet renews it every 10 seconds or so. A lease is tiny, so frequent renewals are cheap; before leases existed, big clusters spent most of their write traffic on node status.
Terminal
kubectl get lease -n kube-node-lease
kubectl get lease worker-a -n kube-node-lease -o jsonpath='{.spec.renewTime}{"\n"}'

The node lifecycle controller (inside kube-controller-manager) watches the leases. If a node has not renewed for --node-monitor-grace-period (50 seconds by default), it sets Ready=Unknown and adds taints:

  • node.kubernetes.io/unreachable:NoExecute (when Ready=Unknown)
  • node.kubernetes.io/not-ready:NoExecute (when Ready=False)

NoExecute evicts Pods that do not tolerate the taint. Kubernetes adds a default toleration with tolerationSeconds: 300 to Pods, so Pods stay on an unreachable node for five minutes before they are evicted and recreated elsewhere. You can shorten or lengthen this per Pod. The stand-in below plays that timeline.

Python Node lifecycle controller: silence, taint, eviction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try changing TOLERATION to 30 and see the Pods leave sooner. That is what tolerationSeconds does in a Pod spec (Taints and Tolerations). Note one important detail: for a StatefulSet, eviction of a Pod on an unreachable node does not mean it is safe to start a replacement, because the old Pod might still be running. That is the reason for the non-graceful shutdown procedure below.

Cordon: stop new Pods

kubectl cordon marks a node unschedulable. Existing Pods keep running, but the scheduler will not place new ones there.

Terminal
kubectl cordon worker-a
kubectl get nodes
TEXT
NAME       STATUS                     ROLES    AGE   VERSION
worker-a   Ready,SchedulingDisabled   <none>   41d   v1.36.1

Behind the scenes it sets spec.unschedulable: true, and the node controller adds the taint node.kubernetes.io/unschedulable:NoSchedule. Use kubectl uncordon worker-a to reverse it.

Drain: cordon plus evict

kubectl drain cordons the node, then evicts every Pod it can, politely, through the Eviction API. Eviction honors PodDisruptionBudgets and the Pod's graceful termination period (Disruptions and Priority). The controller (ReplicaSet, StatefulSet, Job) then creates replacements elsewhere.

Terminal
kubectl drain worker-a --ignore-daemonsets --delete-emptydir-data --timeout=5m

Flags you will meet:

Flag Effect
--ignore-daemonsets Skip DaemonSet Pods (they would be recreated immediately anyway). Without it, drain refuses.
--delete-emptydir-data Allow deleting Pods that use emptyDir volumes (their data is lost).
--force Also delete "bare" Pods that no controller owns. They are not recreated.
--grace-period=N Override each Pod's termination grace period.
--timeout=5m Give up after this long.
--disable-eviction Delete Pods directly and bypass PodDisruptionBudgets. A last resort.
--dry-run=client Show what would happen.

If a PodDisruptionBudget would be violated, drain does not give up: it retries until the replacement Pods are Ready, which is exactly how a budget like "at least 2 web Pods" lets you roll through a cluster one node at a time. The next stand-in models that loop.

Python kubectl drain and a PodDisruptionBudget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Set force=False and the scratch Pod blocks the drain, because nothing would recreate it. Set ignore_daemonsets=False to see the DaemonSet complaint.

For a whole-cluster upgrade procedure built on drain, see Cluster Upgrades and Maintenance.

When a node is shut down

Graceful node shutdown. If the kubelet is configured with shutdownGracePeriod (and shutdownGracePeriodCriticalPods), it notices when the machine begins a systemd shutdown, marks the node not ready, and terminates Pods in order: normal Pods first, then critical ones. Their containers get a chance to run preStop hooks and handle SIGTERM. Without this setting, Pods are killed when the machine goes down.

Non-graceful shutdown. If the machine just vanishes (power loss, a hung hypervisor), the kubelet cannot report anything. Pods look Terminating forever because only the kubelet can confirm their containers stopped, and StatefulSet Pods and their volumes are stuck for safety. When you are certain the machine is off, an operator can add the taint:

Terminal
kubectl taint nodes worker-b node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

This tells the control plane to force-delete the Pods and detach their volumes so replacements can start elsewhere. Never add it to a node that might still be running: two copies of a StatefulSet Pod writing to one disk can corrupt data. Remove the taint when the node is back.

Looking at the container runtime on a node

Because Kubernetes uses a CRI runtime (usually containerd), the docker command shows nothing on a modern node. Node-level debugging uses tools that speak the runtime's own interface, run on the node itself (read-only examples):

Terminal
sudo crictl ps                    # containers the runtime runs (CRI tool, works with containerd and CRI-O)
sudo crictl pods                  # pod sandboxes
sudo crictl logs <container-id>
sudo crictl images                # images cached on the node
sudo nerdctl --namespace k8s.io ps   # Docker-like CLI for containerd

containerd keeps Kubernetes containers in its k8s.io namespace, which is why nerdctl needs the --namespace flag to see them. nerdctl gives containerd a Docker-compatible command line (run, ps, build, compose), useful when you want that experience without Docker Engine. For ordinary work you stay with kubectl; reach for crictl when the API server or kubelet is unhealthy and you must see what the runtime is doing. See also Podman, containerd, Wasm and the OCI Landscape.

Practical checklist

  1. kubectl get nodes, then kubectl describe node X for conditions, taints and allocated resources.
  2. Check the kubelet and runtime logs on the node itself.
  3. For maintenance: cordon, drain --ignore-daemonsets --delete-emptydir-data, do the work, uncordon.
  4. Keep PodDisruptionBudgets on important workloads so a drain can never remove all replicas.
  5. More general debugging steps are in Troubleshooting.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which number should you use when calculating how much a node can host?

    The scheduler uses Allocatable, because the OS, the kubelet and the container runtime need some resources too.
  2. 2. What is the purpose of the Lease objects in the kube-node-lease namespace?

    Leases are tiny objects renewed every few seconds. The node lifecycle controller watches them instead of large status updates.
  3. 3. A node stops renewing its lease. What happens by default after the grace period?

    The controller marks the node Unknown and taints it. Pods carry a default toleration of 300 seconds, so they leave after about five minutes.
  4. 4. What does kubectl cordon worker-a do?

    Cordon only stops scheduling. Drain is cordon plus eviction of the existing Pods.
  5. 5. When is it appropriate to apply the out-of-service taint?

    The taint triggers force deletion and volume detach. On a node that is still running it risks two Pods using the same data.

Summary

  • Kubelets register nodes themselves; the Node object reports addresses, labels, Capacity, Allocatable and conditions.
  • Conditions Ready, MemoryPressure, DiskPressure, PIDPressure and NetworkUnavailable drive taints and evictions. Unknown means the control plane lost contact.
  • Lease heartbeats let the node controller notice silence cheaply; Pods tolerate an unreachable node for 300 seconds by default.
  • cordon stops scheduling, drain evicts Pods politely and respects PodDisruptionBudgets, uncordon restores the node.
  • Use graceful node shutdown for planned reboots and the out-of-service taint only for nodes that are definitely off.

Next, see how to upgrade whole clusters node by node in Cluster Upgrades and Maintenance.