Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 78 ยท Scheduling and Scaling

Node Selectors, Affinity and Anti-Affinity

Control where pods run with nodeSelector, node affinity, pod affinity, anti-affinity and topology spread constraints, with a runnable affinity evaluator.

By default the scheduler treats all nodes that have enough free CPU and memory as equal. That is exactly what you want most of the time, but not always. A database needs a node with fast local SSDs. A machine-learning job needs a GPU. Two replicas of the same web service should never share a node, otherwise one failed machine takes both down. A cache should sit in the same zone as the app that talks to it, to save latency and cross-zone traffic costs.

Kubernetes gives you a ladder of tools for these wishes, from a blunt one-liner to precise, weighted rules. In How the Scheduler Works you saw that scheduling has a filter phase (hard rules) and a score phase (soft rules). Everything in this lesson plugs into one of those two phases: a required rule filters nodes out, a preferred rule only adds to the score.

Four ways to steer a pod

nodeSelector Simple label match. All labels must fit.
Node affinity Rich operators, required or preferred.
Pod affinity Run near pods that match a selector.
Anti-affinity Keep away from matching pods.
1 / 2

Labels on nodes

All of these features read node labels. Kubernetes and your cloud provider already set several. You can list them and add your own:

CONSOLE
$ kubectl get nodes --show-labels
$ kubectl label nodes worker-1 disktype=ssd
$ kubectl label nodes worker-1 disktype-        # a trailing dash removes a label

The well-known labels you will use most are kubernetes.io/hostname (unique per node), topology.kubernetes.io/zone and topology.kubernetes.io/region (set by cloud providers), kubernetes.io/os, kubernetes.io/arch and node.kubernetes.io/instance-type. Labels in general are covered in Labels, Selectors and Annotations.

nodeSelector: the simplest rule

nodeSelector is a map of labels. A node qualifies only if it carries every label in the map with exactly that value. It is a required rule, so a pod nobody can host stays Pending.

YAML
apiVersion: v1
kind: Pod
metadata:
  name: db
spec:
  nodeSelector:
    disktype: ssd
    kubernetes.io/arch: amd64
  containers:
    - name: postgres
      image: postgres:17

This is fine for "must run on SSD nodes". It cannot say "SSD or NVMe", "anything except zone eu-1c" or "prefer SSD but accept anything". For that, you need affinity.

Node affinity

Node affinity lives under spec.affinity.nodeAffinity and comes in two flavors.

requiredDuringSchedulingIgnoredDuringExecution is the hard version. It holds a list of nodeSelectorTerms. The terms are combined with OR: a node needs to satisfy at least one term. Inside a term, all matchExpressions are combined with AND.

preferredDuringSchedulingIgnoredDuringExecution is the soft version. Each entry has a weight from 1 to 100 and a preference (one term). The scheduler adds the weight of every matching entry to the node's score, so heavier wishes count for more.

YAML
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd", "nvme"]
              - key: topology.kubernetes.io/zone
                operator: NotIn
                values: ["eu-1c"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["eu-1b"]
        - weight: 20
          preference:
            matchExpressions:
              - key: cpu-cores
                operator: Gt
                values: ["8"]

The operators are In, NotIn, Exists, DoesNotExist, Gt and Lt. Gt and Lt compare integers, and the label value must parse as one. NotIn and DoesNotExist are how you express anti-affinity to nodes. A node that has no such label at all passes a NotIn test.

The name ends in IgnoredDuringExecution for a reason: the rule is checked only at scheduling time. If you relabel a node afterwards, pods already running there stay put. If you set both nodeSelector and node affinity, the pod must satisfy both.

Try it: evaluate affinity rules yourself

This stand-in is not the real scheduler, but it implements the rules exactly as described above: OR between terms, AND inside a term, and weights added up for the score.

Python Required terms, preferred weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

node-b is filtered out: it has an hdd disk and no GPU label, so no term matches. Of the three survivors, node-c is in zone eu-1b (80 points) while node-a and node-d only earn the 20 points for having more than 8 cores. Change the weights and watch the winner flip. The real scheduler also adds scores from resource balance, image locality and other plugins, so a preferred rule is a nudge, never a promise.

Pod affinity and anti-affinity

Node affinity looks at node labels. Pod affinity looks at which pods already run where. You give a labelSelector that picks other pods and a topologyKey: a node label that defines what "together" means.

  • topologyKey: kubernetes.io/hostname means "the same node".
  • topologyKey: topology.kubernetes.io/zone means "the same zone".

Nodes that share the same value for the topology key form one topology domain. Pod affinity says "put me in a domain that already holds a matching pod". Pod anti-affinity says "put me in a domain that holds none".

YAML
spec:
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: cache
            topologyKey: topology.kubernetes.io/zone
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: web
          topologyKey: kubernetes.io/hostname

This pod tries to land in a zone that already has a cache pod, and it must never share a node with another web pod. The anti-affinity rule is the classic "one replica per node" pattern: put it in a Deployment's pod template, select the Deployment's own label, and use the hostname key. With three nodes and four replicas, the fourth stays Pending, because the rule is required. Use the preferred form if you would rather have it squeeze in.

By default the selector looks at pods in the pod's own namespace. namespaces or namespaceSelector widen that.

Two warnings. First, pod (anti-)affinity is expensive: the scheduler compares every candidate with many existing pods, so Kubernetes documentation advises against it in clusters of several hundred nodes. Second, required anti-affinity makes rolling updates trickier: during an update old and new pods carry the same labels, so with exactly as many nodes as replicas the surge pod has nowhere to go until an old one is gone. Plan for one spare node, or use maxSurge: 0 (see Rolling Updates and Rollbacks).

Topology spread constraints

Anti-affinity is all or nothing: one pod per domain, or none. What you often want is "spread evenly, roughly". Topology spread constraints do exactly that:

YAML
spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: web

For a given domain, skew is the number of matching pods in that domain minus the number in the emptiest domain. maxSkew: 1 allows zones to differ by at most one pod. The scheduler only picks a node whose placement keeps the skew within the limit. whenUnsatisfiable chooses what happens if no node can: DoNotSchedule leaves the pod Pending (a hard rule), ScheduleAnyway places it where the skew is least bad (a soft rule). You can list several constraints, for example one for zones and one for hostnames, and all of them must hold. Newer versions add minDomains (require a minimum number of domains) and matchLabelKeys (for instance pod-template-hash, so that an old ReplicaSet's pods do not distort the count for the new one).

Like affinity, spread constraints apply at scheduling time only. If a Deployment scales down, the remaining pods may end up unbalanced, because nothing moves them. The separate Descheduler project can rebalance.

Try it: watch skew decide

A zone with limited capacity shows the difference between the two policies. Zone C can hold only one pod.

Python maxSkew with DoNotSchedule and ScheduleAnyway

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With DoNotSchedule the sixth pod is Pending: zone C is full with one pod, and putting a third pod in zone A (or a third in B) would make the skew 2. With ScheduleAnyway all six run, ending as 3, 2, 1. This is the practical rule of thumb: use DoNotSchedule when an imbalance is dangerous (a zone outage must not take most replicas), and ScheduleAnyway when availability of capacity matters more.

Choosing the right tool

You want Use
A pod on nodes with a given label nodeSelector
Several allowed values, exclusions, or a soft preference node affinity
Keep two services together for latency pod affinity
One replica per node, strictly pod anti-affinity (required)
Even spread across zones with some slack topology spread constraints
Keep ordinary pods off special nodes taints (next lesson)

Affinity attracts pods to nodes but cannot repel other pods from those nodes. A GPU node labeled gpu=true still accepts any ordinary pod that fits. To reserve nodes you combine affinity with taints, which is the topic of Taints and Tolerations.

What is real and what is the stand-in in this lesson: the YAML and kubectl commands are real Kubernetes, and none of it runs on this site because there is no cluster here. The Python programs re-implement the matching and skew arithmetic so you can experiment, but the real scheduler evaluates many more plugins.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A pod has a required node affinity with two nodeSelectorTerms. How are they combined?

    nodeSelectorTerms are ORed. Inside one term, the matchExpressions are ANDed.
  2. 2. What does topologyKey: kubernetes.io/hostname mean in a pod anti-affinity rule?

    Each node has a unique hostname label, so every node is its own topology domain. Anti-affinity then allows at most one matching pod per node.
  3. 3. With maxSkew: 1 and whenUnsatisfiable: DoNotSchedule, zone A holds 2 matching pods, zone B holds 2 and zone C (full) holds 1. What happens to the next pod?

    Zone C cannot take it, and placing it in A or B would give a skew of 2, which breaks maxSkew. DoNotSchedule is a hard rule, so the pod waits.
  4. 4. Which statement about preferred affinity is true?

    Preferred rules are scoring hints. Pods still schedule on other nodes, and rules are ignored during execution, so running pods are not moved.

Summary

  • nodeSelector is the simplest rule: all listed node labels must match.
  • Node affinity adds operators (In, NotIn, Exists, DoesNotExist, Gt, Lt), OR between terms, AND inside a term, and a required or preferred flavor.
  • Pod affinity and anti-affinity use a labelSelector plus a topologyKey that defines what "together" means: node, zone or region.
  • Topology spread constraints keep the count of matching pods across domains within maxSkew.
  • Required rules filter nodes and can leave pods Pending; preferred rules only add to the score.
  • All of these apply at scheduling time only; use taints to keep other pods off special nodes.

Next, learn how nodes push pods away in Taints and Tolerations.