Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 78 ยท Scheduling and Scaling

PodDisruptionBudgets, Priority and Eviction

Keep services up during drains and upgrades with PodDisruptionBudgets, and learn priority classes, preemption and node-pressure eviction with runnable simulations.

Pods do not live forever. Nodes get upgraded, autoscalers remove machines, hardware breaks, and a busy node runs out of memory. Some of these events are planned and polite, others are sudden. Kubernetes gives you tools for both kinds: a PodDisruptionBudget tells the platform how many replicas of your service may be down at once during planned work, and priority classes tell the scheduler and the kubelet which pods matter most when there is not enough room.

This lesson ends the scheduling chapter by showing how these pieces decide who stays and who goes.

Two kinds of disruption

Voluntary Drains, upgrades, autoscaler scale-down.
Involuntary Hardware failure, kernel panic, VM deleted.
Pressure The kubelet evicts pods when a node runs low.
1 / 2

Voluntary and involuntary disruptions

A voluntary disruption is one that an administrator or a controller starts on purpose: draining a node for a kernel update, upgrading the cluster, the Cluster Autoscaler removing an underused node, or a Deployment rolling out a new version. These can be coordinated, so Kubernetes lets you say how much is acceptable.

An involuntary disruption just happens: a machine dies, the hypervisor is rebooted, a node is out of memory, a network partition makes a node unreachable. Nothing can ask permission first, and a PodDisruptionBudget cannot prevent any of it. Defend against these with several replicas, spread across nodes and zones (see Node Selectors, Affinity and Anti-Affinity), plus good probes.

PodDisruptionBudgets

A PodDisruptionBudget (PDB) selects a group of pods with a label selector and states either how many must stay available (minAvailable) or how many may be unavailable (maxUnavailable). Use only one of the two. Both accept an absolute number or a percentage.

YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: web

With five web pods this allows one voluntary eviction at a time (5 healthy minus 4 required). If one pod is already unhealthy or in the middle of being replaced, the number of allowed disruptions falls to zero. You can read it from the PDB's status:

CONSOLE
$ kubectl get pdb web
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web    4               N/A               1                     3d

Important details:

  • A PDB only applies when pods are removed through the Eviction API. kubectl drain, the Cluster Autoscaler and Karpenter use it. A plain kubectl delete pod, a node crash or a Deployment's own rolling update do not (the rolling update has its own maxUnavailable; see Rolling Updates and Rollbacks).
  • A refused eviction returns HTTP 429 (Too Many Requests), and the caller retries. A drain therefore simply waits until the budget allows another pod to go.
  • minAvailable equal to the number of replicas (or maxUnavailable: 0) means no pod can ever be evicted. The node can never be drained, and cluster upgrades hang. This is the most common PDB mistake. Prefer maxUnavailable: 1 or a percentage.
  • A PDB for a single-replica workload with minAvailable: 1 blocks all drains. Either run two or more replicas or accept the downtime.
  • The selector should match the pods of one workload, not everything in the namespace.
  • unhealthyPodEvictionPolicy: AlwaysAllow lets the platform evict pods that are not Ready even if the budget would say no, which avoids a broken pod blocking a drain forever.

Draining a node

kubectl drain is the polite way to empty a node:

CONSOLE
$ kubectl drain node-a --ignore-daemonsets --delete-emptydir-data
node/node-a cordoned
evicting pod default/web-1
evicting pod default/web-2
error when evicting pods/"web-2" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/web-1 evicted
pod/web-2 evicted
node/node-a drained

It first cordons the node (adds the node.kubernetes.io/unschedulable taint described in Taints and Tolerations), then evicts each pod through the Eviction API. DaemonSet pods are skipped with --ignore-daemonsets, because they would be recreated at once. Pods using emptyDir lose that data, which --delete-emptydir-data acknowledges. Pods not managed by any controller are not recreated, so drain refuses to touch them unless you pass --force. Afterwards kubectl uncordon node-a makes the node schedulable again.

Try it: a drain that honors a budget

The simulation drains node-a with two pods on it. Each evicted pod is replaced on another node, and the replacement takes two ticks to become Ready. The PDB demands at least four healthy pods out of five.

Python Draining with minAvailable: 4

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At t=0 there are five healthy pods and one disruption is allowed, so web-1 is evicted. At t=1 its replacement is not Ready yet, only four pods are healthy, the budget allows nothing and the eviction of web-2 is refused with a 429. At t=2 the replacement web-6 becomes Ready, the budget opens again, and web-2 goes. The drain finishes one tick later. The service never dropped below four pods, which is exactly the promise of the PDB. Try MIN_AVAILABLE = 5 and the drain never finishes: the loop runs out of ticks while every eviction is refused.

Priority classes

When the cluster is full, not all pods are equally important. A PriorityClass assigns an integer priority to pods; higher means more important.

YAML
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Customer-facing services"

A pod opts in with priorityClassName: business-critical in its spec. Pods without a class get priority 0 (or the class marked globalDefault, of which there can be only one). Kubernetes ships two classes for its own components, system-cluster-critical and system-node-critical, with values around two billion; user-defined classes must stay below one billion. Priority has two effects.

  1. Queue order. The scheduler takes higher-priority pending pods first.
  2. Preemption. If a pending pod fits on no node, the scheduler looks for a node where evicting some lower-priority pods would make room. It picks the node whose victims are the least important (it prefers a choice that violates the fewest PDBs, then the one whose most important victim has the lowest priority, then the lowest total priority, then the fewest victims). The victims are given their normal termination grace period, the pod is nominated for that node, and it is scheduled once the room is free. With preemptionPolicy: Never, a pod still gets queue priority but never evicts anyone.

PDBs are respected on a best-effort basis during preemption; if no other option exists the scheduler may still violate them.

Try it: preemption decides on victims

A pending pod needs 3000 m CPU. Two nodes are full; the simulation finds which running pods would have to leave, node by node, and chooses the cheapest set.

Python Which pods get preempted?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Either node could host the new pod, each at the cost of two victims. Node 1's victims are the priority-100 batch pods, node 2's are two priority-1000 web pods, so the simulation nominates node 1 and the web pods are spared. The design lesson: give batch and background work a low priority and customer-facing services a high one, and the cluster sacrifices the right pods when it is squeezed. Do not make everything "critical": a priority scheme where every pod has the highest class has no information in it.

A practical trick builds on this: run a few low-priority "pause" pods that reserve spare room. When a real pod needs the space, the placeholders are preempted instantly, and the node autoscaler then replaces the room in the background.

Node-pressure eviction

Even without any scheduler involvement, a node can run short of resources. The kubelet watches eviction signals such as memory.available, nodefs.available, imagefs.available, nodefs.inodesFree and pid.available. Each can have a soft threshold with a grace period and a hard threshold that acts immediately. A typical hard default is memory.available<100Mi. When a threshold is crossed, the node reports a pressure condition (MemoryPressure, DiskPressure, PIDPressure), the matching NoSchedule taint appears so no new pods arrive, and the kubelet starts evicting pods until the signal recovers.

Which pods go first? The kubelet ranks them by:

  1. whether the pod's usage of the starved resource exceeds its requests (those go first),
  2. then by pod priority (lower first),
  3. then by how far usage is above requests.

This is why setting requests honestly matters: a pod that stays within its requests is the last to be evicted, which links to the QoS classes in Resource Requests and Limits. Node-pressure eviction is not the same as the kernel's OOM killer. The OOM killer acts inside the kernel when memory is already gone and kills a container process (you see OOMKilled). The kubelet tries to act earlier and more gracefully. Evicted pods end in phase Failed with reason Evicted; a controller creates replacements, but a bare pod is simply gone. Node-pressure eviction ignores PDBs, because it is an emergency, not a planned operation.

What is real and what is the stand-in: all YAML and kubectl output is real but read-only on this site (the drain output is typical, not captured from a specific cluster). The Python programs model the budget arithmetic and victim selection with one resource (CPU) and simplified tie-breaking.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which event is NOT prevented by a PodDisruptionBudget?

    PDBs only constrain voluntary evictions made through the Eviction API. A crashed node is an involuntary disruption.
  2. 2. A Deployment has 3 replicas and a PDB with minAvailable: 3. What happens when you drain a node that runs one of them?

    With minAvailable equal to the replica count, allowed disruptions is always 0, so every eviction is refused (429) and the drain hangs.
  3. 3. What does preemption do?

    When a pending pod fits nowhere, the scheduler may evict lower-priority pods on a chosen node to make room.
  4. 4. Under node memory pressure, which pods does the kubelet evict first?

    Eviction order is: usage above requests, then lower priority, then the size of the overshoot. Pods staying inside their requests are protected longest.

Summary

  • Voluntary disruptions (drain, upgrade, autoscaling) can be controlled; involuntary ones (crashes) need replicas and spreading.
  • A PodDisruptionBudget (minAvailable or maxUnavailable) limits evictions made through the Eviction API; refused evictions return 429 and drains wait.
  • Never set a budget that allows zero disruptions, or drains and upgrades hang.
  • PriorityClasses order the scheduling queue and enable preemption of lower-priority pods; preemptionPolicy: Never opts out of evicting others.
  • The kubelet evicts pods under node pressure, ranking by overshoot of requests, priority and usage; this ignores PDBs.
  • Honest requests plus sensible priorities decide who survives a squeeze.

Next, the security chapter begins with Authentication and RBAC.