Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 78 ยท Scheduling and Scaling

How the Scheduler Works

See how kube-scheduler picks a node for every pod: the queue, filtering and scoring phases, resource fit, nodeName, binding, and why pods stay Pending.

When you create a pod, nobody tells it where to run. A pod starts life without a node, and a control plane component called the kube-scheduler watches for such pods, chooses a node for each, and writes that choice into the pod. From then on, the kubelet on that node takes over. The scheduler never starts containers itself; its entire job is decide where.

This lesson walks through how that decision is made. You will see the scheduling queue, the two big phases (filter, then score), the way resource requests drive placement, how nodeName bypasses the scheduler, and how to read the message when a pod stays Pending. You met the scheduler in cluster architecture; the next lessons add rules on top: node selectors and affinity and taints and tolerations. The scheduler cannot run on this site, so the examples are a Python stand-in with the same two phases.

Two questions per pod

Filter Which nodes can run this pod at all?
Score Of those, which one is best?

Filtering is yes or no. Scoring is a number from 0 to 100.

1 / 2

The scheduling queue

The scheduler holds pods that have no spec.nodeName in a queue and takes them one at a time. There are three internal parts:

  • the active queue: pods ready to be tried now, ordered by priority (priority classes) and then by arrival;
  • the backoff queue: pods that failed recently and wait a short, growing delay before another attempt;
  • the unschedulable set: pods that cannot fit anywhere right now. They are re-queued automatically when something changes that might help, such as a new node joining, a pod deleting or a label changing.

Each pod goes through a scheduling cycle (choose a node), followed by a binding cycle (record the choice). Scheduling cycles run one after another, which keeps decisions consistent. Binding cycles run asynchronously so the scheduler can start on the next pod while the API server is still saving the last result.

Step 1: filtering

Filtering removes every node that cannot run the pod. A node survives only if all filter plugins accept it. The main ones:

Check What it rejects
Node unschedulable Cordoned nodes (kubectl cordon)
Node resources fit Nodes without enough free CPU, memory, ephemeral storage or pod slots for the pod's requests
Node name Nodes other than the one in spec.nodeName
Node ports Nodes where a requested hostPort is already taken
Node selector and affinity Nodes whose labels do not match
Taints and tolerations Nodes with a taint the pod does not tolerate
Pod topology spread / pod anti-affinity Nodes that would violate spread or anti-affinity rules
Volume binding, volume zone Nodes where the pod's volumes cannot attach (StorageClasses)

The most important rule for beginners: the scheduler fits requests, not actual usage. A node with 4 CPUs where pods have requested 3.8 CPUs is nearly full, even if those pods are idle at 5% CPU. A pod with no requests counts as zero and fits anywhere, which is why pods without requests lead to overcrowded nodes (requests and limits). For a pod with init containers, the effective request is the larger of the biggest init container and the sum of the app containers.

Step 2: scoring

If several nodes survive, scoring ranks them. Each scoring plugin gives each node 0 to 100 points, each score is multiplied by the plugin's weight, and the sums are added. The highest total wins; ties are broken randomly. Default plugins include:

  • NodeResourcesFit with the LeastAllocated strategy: prefers nodes with the most free resources after placing the pod. That spreads load out. (You can configure MostAllocated instead to pack nodes tightly, useful with a cluster autoscaler.)
  • NodeResourcesBalancedAllocation: prefers nodes where CPU and memory usage end up balanced, so you do not strand memory on a node whose CPU is full.
  • ImageLocality: prefers nodes that already hold the container image.
  • Pod topology spread, inter-pod affinity, node affinity and taint toleration: soft preferences ("prefer this zone", "prefer not to share a node").

In very large clusters the scheduler does not score every node. The percentageOfNodesToScore setting (by default an adaptive value that shrinks as the cluster grows) stops the search once enough feasible nodes are found, trading a perfect choice for speed.

Step 3: reserve and bind

After picking a node, the scheduler assumes the pod is there (so the next pod sees the resources as taken), runs reserve and permit plugins, and then in the binding cycle creates a Binding object through the API server. That sets spec.nodeName on the pod. The kubelet on that node watches for pods assigned to it, pulls the image and starts the containers. If binding fails, the assumption is rolled back and the pod returns to the queue.

flowchart LR
    Q["Queue<br/>pod without node"] --> F["Filter<br/>drop unfit nodes"]
    F -->|"some nodes left"| S["Score<br/>rank the rest"]
    F -->|"none left"| U["Unschedulable<br/>retry later"]
    S --> B["Bind<br/>set spec.nodeName"]
    B --> K["Kubelet<br/>starts the pod"]

Filter, score, bind: the whole job of kube-scheduler.

Writing a scheduler

The stand-in below implements both phases with the real ideas: filters that record a reason for every rejected node, and LeastAllocated plus BalancedAllocation scoring. Ties go to the first node by name here (the real scheduler picks randomly among the best, so that load does not pile onto one node).

Python A scheduler with filter and score phases

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

node-d is cordoned and node-c has a taint the pod does not tolerate, so they are filtered out. Of the two survivors, node-a is half-busy on CPU but has plenty of memory, while node-b is almost out of memory. node-a scores 125.0 against 112.5 for node-b, because it ends up with more free capacity and a better CPU-to-memory balance. Try giving the pod "nodeSelector": {"disk": "ssd"} and see how the candidate list changes.

nodeName: skipping the scheduler

If you set spec.nodeName yourself, the scheduler ignores the pod completely, and the kubelet on that node simply tries to run it:

YAML
spec:
  nodeName: node-b
  containers:
    - name: app
      image: example/app:1.0

None of the filters apply: no resource check, no taints, no selectors. If node-b does not exist or is full, the pod fails there instead of waiting. It is occasionally useful for debugging or custom schedulers, but for normal workloads use nodeSelector or affinity so the scheduler still validates the choice.

Why is my pod Pending?

When no node passes the filters, the scheduler records a FailedScheduling event on the pod, summarizing why every node was rejected:

Terminal
kubectl describe pod web-7
# Events:
#   Warning  FailedScheduling  0/4 nodes are available:
#     1 Insufficient cpu, 1 node(s) were unschedulable,
#     1 node(s) had untolerated taint {gpu: true},
#     1 node(s) didn't match Pod's node affinity/selector.

Read it as a tally that adds up to the node count. Insufficient cpu or Insufficient memory means your requests do not fit anywhere (lower them, add nodes, or let the cluster autoscaler add one). untolerated taint means a toleration is missing. didn't match node affinity/selector points at a label typo. Unbound volume messages point at storage (PersistentVolumes). The stand-in below produces the same style of summary and fills a tiny cluster until a pod no longer fits.

Python Filling a cluster until pods stay Pending

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that pod-7 stays Pending even though the cluster as a whole still has some free CPU: no single node has 2,500 m left. Total free capacity does not matter; only per-node capacity does. That is a classic reason to keep pods small and to leave headroom.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the scheduler compare against a node's capacity when filtering for resources?

    The scheduler fits requests against the node's allocatable resources. Real usage and limits are not part of the decision, so pods without requests count as zero.
  2. 2. What happens in the scoring phase?

    Filtering removes unfit nodes. Scoring ranks the remaining ones with weighted plugin scores, and the best node is selected.
  3. 3. A pod sets spec.nodeName to a full node. What does the scheduler do?

    Setting nodeName bypasses the scheduler and all its filters. The kubelet on the named node handles the pod and can reject it.
  4. 4. The event says: 0/4 nodes are available: 3 Insufficient memory, 1 node(s) were unschedulable. What is the most useful first step?

    Three nodes were rejected for lack of memory against the pod's request, so the request has to fit or capacity has to grow. The cordoned node is a separate, deliberate condition.
  5. 5. Why can a pod stay Pending although the cluster has enough total free CPU?

    A pod must fit on one node. Fragmented free capacity across many nodes cannot satisfy a single large request.

Summary

  • kube-scheduler takes unscheduled pods from a queue, picks a node, and records it in spec.nodeName; the kubelet does the rest.
  • Filtering removes nodes that cannot run the pod (resources, taints, selectors, ports, volumes, cordons); scoring ranks the rest and ties break randomly.
  • Placement is based on requests, not usage; pods without requests make nodes look emptier than they are.
  • Binding is asynchronous; assumed pods keep later pods from over-filling a node.
  • spec.nodeName skips the scheduler and every check.
  • FailedScheduling events list why each node was rejected, and free capacity must exist on a single node.

Next: Node Selectors, Affinity and Anti-Affinity