Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 78 ยท Scheduling and Scaling

Taints and Tolerations

Learn how node taints repel pods and tolerations let them in: NoSchedule, PreferNoSchedule, NoExecute, dedicated nodes and built-in taints, with runnable matchers.

Affinity lets a pod say "I want to run over there". A taint works in the opposite direction: a node says "stay away, unless you have a good reason". The good reason is a toleration on the pod.

Why do you need this? Imagine a few expensive GPU nodes. If you only label them and give your ML pods a node selector, ordinary web pods can still land on the GPU nodes whenever they have free CPU, and then the GPUs sit idle while the nodes are "full". Or imagine the control plane nodes: you do not want your applications there. Taints solve both. Kubernetes itself also uses taints to react to node problems: a node that stops responding gets a taint, and that is what drives pod eviction.

Taint on the node, toleration on the pod

Taint Set on a node. Repels pods.
Toleration Set on a pod. Allows it past one taint.
Not a magnet A toleration permits a node. It does not choose it.
1 / 2

The anatomy of a taint

A taint has a key, an optional value and an effect, written key=value:effect. You add and remove taints with kubectl:

CONSOLE
$ kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule
node/gpu-1 tainted
$ kubectl describe node gpu-1 | grep Taints
Taints:             dedicated=gpu:NoSchedule
$ kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule-     # trailing dash removes it
node/gpu-1 untainted

The three effects differ in how hard they push:

  • NoSchedule: the scheduler will not place a new pod on the node unless the pod tolerates the taint. Pods that are already running are left alone.
  • PreferNoSchedule: a soft version. The scheduler avoids the node if there is any alternative, but will use it when nothing else fits.
  • NoExecute: the strictest one. New pods without a toleration are kept out and running pods without a toleration are evicted from the node.

A node can carry many taints. For a pod to be allowed, it must tolerate every NoSchedule and NoExecute taint on the node. Each untolerated PreferNoSchedule taint only lowers the node's score.

Writing tolerations

Tolerations go in the pod spec (in a Deployment, that is the pod template):

YAML
spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 60
  containers:
    - name: trainer
      image: registry.example.com/trainer:3

A toleration matches a taint when:

  • the key is equal, and
  • the effect is equal (an empty effect matches every effect), and
  • with operator: Equal (the default) the value is equal; with operator: Exists the value is ignored and must be left out.

Two special cases are worth remembering. An Exists toleration with an empty key tolerates every taint, which is what you want for something like a node-level monitoring agent, and is dangerous elsewhere. And tolerationSeconds only makes sense with NoExecute: the pod is allowed to remain on the node for that many seconds after the taint appears, and is then evicted. Without tolerationSeconds, a matching toleration lasts forever.

Try it: a toleration matcher

This is the matching logic from the rules above, tested against four nodes and five pods. The verdict is the strongest blocking effect among the taints the pod does not tolerate.

Python Which pod may run on which node?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what the output teaches. The gpu job is allowed on gpu-node, but its toleration does not allow the control plane or the dead node, because each taint needs its own matching toleration. The "wrong value" pod is blocked even though its key matches. And the Exists pod with no key walks past everything, which is why such a toleration should be rare.

Dedicated nodes: taint plus affinity

A common mistake is to think a toleration reserves a node for a pod. It does not. A toleration only permits scheduling; the gpu job above may still land on any untainted node. To dedicate nodes to a workload you need both halves:

  1. Taint the nodes (dedicated=gpu:NoSchedule) so ordinary pods stay away.
  2. Give the node a label (dedicated=gpu) and add node affinity or a nodeSelector to the GPU pods, so they go only there.
  3. Add the matching toleration to the GPU pods so they are allowed in.

Managed clusters often do this for you: you create a node pool with taints and labels, and your pod specs only need the toleration and selector. Many admission controllers can even inject the toleration automatically for a given namespace.

Taints that Kubernetes adds itself

The node lifecycle controller and the kubelet add taints when a node is unhealthy, which is how Kubernetes turns "this node has a problem" into scheduling decisions:

Taint Effect When
node.kubernetes.io/not-ready NoExecute Node condition Ready is false
node.kubernetes.io/unreachable NoExecute The control plane lost contact with the node
node.kubernetes.io/memory-pressure NoSchedule The node is short of memory
node.kubernetes.io/disk-pressure NoSchedule The node is short of disk
node.kubernetes.io/pid-pressure NoSchedule The node has too many processes
node.kubernetes.io/network-unavailable NoSchedule The node network is not configured yet
node.kubernetes.io/unschedulable NoSchedule The node is cordoned

kubectl cordon and kubectl drain use the last one. Control plane nodes in most distributions carry node-role.kubernetes.io/control-plane:NoSchedule, which is why your pods do not appear there on a normal cluster. On a single-node learning cluster such as one created by kind or minikube that taint is removed, otherwise nothing could run.

DaemonSets get tolerations for several of these built-in taints automatically, so node-level agents (log shippers, CNI plugins) keep running on unhealthy nodes. See DaemonSets.

NoExecute and tolerationSeconds in practice

When a node becomes unreachable, you might expect every pod on it to move away immediately. That would be a bad idea: the node might only be rebooting or have a brief network blip, and restarting hundreds of pods elsewhere is expensive. So Kubernetes adds a default toleration to every pod (through an admission plugin) for not-ready and unreachable with tolerationSeconds: 300. A pod is evicted five minutes after its node is declared bad, and its controller (Deployment, StatefulSet, ...) creates a replacement elsewhere.

You can override this per workload. A stateless web service that recovers quickly might use 30 seconds, so that traffic moves away sooner; a pod holding a large local cache might use more, or no limit at all. Only the pod's own spec matters:

Python When is each pod evicted?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pod with no tolerationSeconds ("tolerates forever") is the dangerous one. It never leaves the dead node, so for a StatefulSet pod that may be deliberate (it avoids two copies of a database running at once), but for a Deployment pod it means lost capacity. Note also that a StatefulSet pod stuck on an unreachable node is not replaced automatically even after eviction, because Kubernetes cannot be sure the old pod has really stopped. See StatefulSets.

Taints, affinity and the scheduler

In the scheduler's filter phase, the taint check rejects nodes with untolerated NoSchedule or NoExecute taints, and node affinity rejects nodes that fail its required terms. Both must pass. In scoring, untolerated PreferNoSchedule taints lower a node's rank, and preferred affinity raises it. If you think of the two features together: affinity is for the pod's wishes, taints are for the node's restrictions.

A short checklist for debugging a Pending pod: run kubectl describe pod, read the Events section, and look for messages such as 0/5 nodes are available: 2 node(s) had untolerated taint {dedicated: gpu}, 3 node(s) didn't match Pod's node affinity/selector. The message counts nodes per failed reason, so you can tell which rule is the blocker.

What is real and what is the stand-in: the kubectl taint commands and manifests above are real, read-only examples because this site has no cluster. The Python matcher is a faithful re-statement of the matching rules, without the rest of the scheduler.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A node has the taint dedicated=gpu:NoSchedule. A pod with a matching toleration is created. Which statement is true?

    A toleration only permits scheduling on the tainted node. To force the pod there you also need a nodeSelector or node affinity.
  2. 2. Which effect evicts pods that are already running on the node?

    NoExecute applies to running pods too. NoSchedule and PreferNoSchedule only influence the placement of new pods.
  3. 3. What does a toleration with operator: Exists and no key tolerate?

    An empty key with Exists matches all keys, and an empty effect matches all effects. This is typical for node agents but should be avoided for ordinary workloads.
  4. 4. A node becomes unreachable. By default, when are ordinary pods evicted?

    An admission plugin adds tolerations for not-ready and unreachable with tolerationSeconds 300 to every pod, so the eviction waits five minutes unless you override it.

Summary

  • A taint (key=value:effect) on a node repels pods; a toleration on a pod permits one or more taints.
  • NoSchedule blocks new pods, PreferNoSchedule is a soft preference, NoExecute also evicts running pods.
  • A pod must tolerate every blocking taint on a node, and tolerations never attract pods.
  • To dedicate nodes, combine a taint with a label plus node affinity or nodeSelector.
  • Kubernetes adds taints for unready, unreachable, pressure and cordoned nodes; default tolerations delay eviction by 300 seconds.

Next: how to divide a cluster between teams in Namespaces, Quotas and Limit Ranges.