Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 78 ยท Scheduling and Scaling

Static Pods and Manual Scheduling

Learn how the kubelet runs static Pods from a manifest directory, what mirror Pods are, how nodeName bypasses the scheduler, and why a running Pod cannot be moved.

Normally a Pod's life goes like this: you send it to the API server, the scheduler picks a node, and the kubelet on that node starts it. Two special cases skip part of that path. A static Pod is started by the kubelet alone, straight from a file on disk, with no scheduler and no controller. Manual scheduling means you choose the node yourself by writing spec.nodeName, so the scheduler is never asked.

Both are rarely used in applications, but both are very much alive inside every cluster. If you have ever listed kube-system and seen Pods called etcd-cp-1 or kube-apiserver-cp-1, you have met static Pods. This lesson explains how they work, how manual placement works, and why a Pod can never be moved once it is running. There is no cluster on this site, so manifests and kubectl commands are read-only; the two runnable examples are small Python stand-ins.

Static Pods

A static Pod is defined by a manifest file in a directory that the kubelet watches. The directory is set in the kubelet configuration (staticPodPath); on kubeadm clusters it is /etc/kubernetes/manifests. The kubelet:

  1. reads every Pod manifest in that directory, and starts and supervises those Pods itself, restarting containers as for any Pod;
  2. watches the directory, so that adding, editing or removing a file creates, restarts or stops the Pod;
  3. tries to create a mirror Pod in the API server for each one, so that kubectl get pods shows it.

The name of a static Pod is the name in the file plus the node name: a manifest called etcd on node cp-1 becomes the Pod etcd-cp-1.

Terminal
# on a control plane node
ls /etc/kubernetes/manifests
TEXT
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

This is how a kubeadm control plane bootstraps itself. The API server cannot be scheduled by a scheduler that needs the API server; so the kubelet runs them from disk, and once they are up, the rest of the cluster works normally (Cluster Architecture). It is also why upgrading a control plane component means editing a file (Cluster Upgrades and Maintenance).

A static Pod manifest

It is an ordinary Pod manifest. Save it into the manifest directory of one node:

YAML
# /etc/kubernetes/manifests/node-exporter.yaml
apiVersion: v1
kind: Pod
metadata:
  name: node-exporter
  labels:
    app: node-exporter
spec:
  hostNetwork: true
  containers:
    - name: exporter
      image: quay.io/prometheus/node-exporter:v1.8.2
      args: ["--path.rootfs=/host"]
      volumeMounts:
        - name: root
          mountPath: /host
          readOnly: true
  volumes:
    - name: root
      hostPath:
        path: /

Within seconds kubectl get pods -A shows node-exporter-<node-name> in the default namespace. You do not apply it with kubectl; the kubelet picks it up. (The kubelet can also fetch a manifest from a URL, staticPodURL, but a file is the norm.)

Mirror Pods: a read-only reflection

The Pod you see through kubectl is only a mirror: an API object created by the kubelet so that the rest of the cluster knows the Pod exists and how much resource it uses. The API server cannot control it:

  • kubectl delete pod etcd-cp-1 deletes only the mirror. The kubelet immediately re-creates it, and the containers keep running.
  • You cannot change the Pod through the API. Edit the file instead.
  • To stop a static Pod you delete the file (or move it out of the directory).
  • Static Pods cannot reference ConfigMaps, Secrets, ServiceAccounts or other API objects, because the kubelet must be able to run them even when the API server is down. They also cannot have ephemeral containers.

The stand-in below models a kubelet loop. Watch what each action does:

Python A kubelet with a static Pod directory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When to use static Pods

  • Control plane components, as kubeadm does.
  • Node-local agents you want even if the API server is down. In most clusters a DaemonSet is better: it is managed through the API, rolls out updates and works with RBAC and ConfigMaps (DaemonSets).
  • Small edge or lab setups where one node runs a few fixed Pods and no control plane exists. (Single-node kubelet-only setups exist, but have no scheduling, Services or controllers.)

Static Pods are updated by editing files, so manage those files with configuration management or your image pipeline, and keep them under version control. On managed clusters (EKS, GKE, AKS) you cannot touch the control plane's files at all.

Manual scheduling with nodeName

Every Pod has an optional field spec.nodeName. When the scheduler sees a Pod without it, it picks a node and writes the name ("binding"). When the field is already set, the scheduler ignores the Pod, and the kubelet on that node starts it:

YAML
apiVersion: v1
kind: Pod
metadata:
  name: pinned
spec:
  nodeName: worker-b
  containers:
    - name: app
      image: nginx:1.27

What this means in practice:

  • No filtering. The scheduler's checks (resources, taints, affinity) are skipped. If worker-b is out of memory, the Pod is started anyway and may be rejected or evicted by the kubelet (you will see OutOfmemory, OutOfcpu or NodeAffinity errors from the kubelet itself).
  • No tolerations needed for scheduling. A NoSchedule taint does not stop it (a NoExecute taint still will evict it).
  • A typo is a stuck Pod. If the node does not exist, the Pod stays Pending forever with no events from a scheduler, because nothing is trying.
  • Not portable. Node names change between clusters and when nodes are replaced.

Legitimate uses: debugging the scheduler, running something on a specific node while you investigate, and test setups. For normal placement use nodeSelector, affinity, taints and topology spread constraints, which keep the scheduler involved and checked (Node Selectors and Affinity, Taints and Tolerations). To pin a Pod to a node in a way that survives node replacement, use the label kubernetes.io/hostname in a nodeSelector; unlike nodeName, the scheduler still checks that the node has room.

What the scheduler actually does

For a Pod without nodeName the scheduler runs a queue, then filter (drop nodes that cannot run the Pod), then score (rank the survivors), then bind (write nodeName through the binding API). The Scheduler walks through that with a runnable model; here the key idea is simply that binding is the only thing it writes. Setting nodeName yourself is the same write, made by a human.

You can also run a second scheduler (or a profile of the default one) and pick it per Pod with spec.schedulerName. Pods that name a scheduler that does not exist stay Pending.

Moving a running Pod: you can't

A Pod is bound to a node for its whole life. The nodeName field is immutable once set, and the Binding API refuses a second binding. If you try to change it:

  • Binding twice returns 409 Conflict ("already assigned").
  • Editing spec.nodeName of a running Pod is rejected, because Pod updates may only change a short list of fields (container images, tolerations and a few others). The message is abridged in the model below.

To "move" a Pod you delete it and create a new one, which has a new IP address, a new UID and a fresh start. Controllers do exactly that when they replace Pods; for a bare Pod it is your job:

Terminal
kubectl get pod web -o yaml > web.yaml      # save it
# edit web.yaml: change spec.nodeName (or remove it and use a nodeSelector),
# and remove status and runtime-only metadata fields
kubectl replace --force -f web.yaml         # delete and recreate in one step

For a Pod owned by a Deployment, never move individual Pods: cordon the node, add an affinity to the template, or drain it, and the ReplicaSet will start replacements elsewhere (Nodes, Deployments).

Python Binding is a one-time write

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Who creates and supervises a static Pod?

    Static Pods are started by the kubelet alone, from files on the node. The API server only shows a mirror.
  2. 2. You run kubectl delete pod etcd-cp-1 for a static Pod. What happens?

    The API object is just a reflection. To stop a static Pod, remove its manifest file.
  3. 3. What happens to a Pod that has spec.nodeName set when it is created?

    A Pod that already has a node is not scheduled. The kubelet starts it, and filters such as resources or taints are not applied by the scheduler.
  4. 4. Which is a safer way than nodeName to keep a Pod on one particular node?

    A nodeSelector keeps the scheduler involved, so it still checks capacity and taints.
  5. 5. How do you move a running bare Pod to a different node?

    Placement is fixed for a Pod's life. Moving means recreating, and the new Pod has a new identity and IP address.

Summary

  • A static Pod is run by the kubelet from a file in staticPodPath (/etc/kubernetes/manifests with kubeadm); the API shows a read-only mirror named <pod>-<node>.
  • Control plane components on kubeadm clusters are static Pods. For ordinary node agents prefer a DaemonSet.
  • Setting spec.nodeName skips the scheduler entirely, and with it all filtering. Use it for debugging, and use selectors and affinity for real placement.
  • Binding happens once. Pods cannot be moved; they are deleted and recreated.

Next: Node Selectors and Affinity shows how to steer the scheduler instead of bypassing it.