Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 78 ยท Workloads

DaemonSets

Learn how a DaemonSet runs one pod per node for log shippers and monitoring agents, how node selection and tolerations work, and how its updates roll out.

Some software does not belong to the application; it belongs to the machine. A log shipper has to read the log files of every node. A monitoring agent has to report the CPU of every node. A network plugin has to configure networking on every node. For these you do not want "three copies somewhere"; you want exactly one copy on each node, and a new copy automatically whenever a node joins.

That is what a DaemonSet does. Where a Deployment says "keep N pods", a DaemonSet says "keep one pod on every (eligible) node". Add a node and its pod appears; remove the node and the pod is garbage collected. This lesson covers the manifest, how nodes are selected (node selectors, affinity and taints), typical use cases, update strategies, and a Python stand-in that places pods on eligible nodes.

One pod per node

Node joins A pod is created for it.
Node leaves Its pod is removed.
Subset A selector can limit the nodes.
1 / 2

A DaemonSet manifest

The manifest looks like a Deployment without replicas. The example below is read-only, since there is no cluster on the site; it runs a hypothetical log collector.

YAML
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: log-agent
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: log-agent
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: agent
          image: registry.example.com/log-agent:1.4
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 128Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Three things stand out:

  • No replicas. The number of pods is the number of eligible nodes.
  • hostPath volumes are common here, because the whole point is to look at the node: its log directory, its container runtime socket, /proc, /sys. These are powerful, so only trusted agents should be given them (see pod security).
  • Tolerations let the agent run on tainted nodes, such as the control plane, where ordinary pods are kept out (see the next section).

Useful read-only commands:

Terminal
kubectl get daemonset -n kube-system
kubectl get pods -n kube-system -l app=log-agent -o wide
kubectl rollout status daemonset/log-agent -n kube-system
kubectl describe daemonset log-agent -n kube-system

kubectl get daemonset shows DESIRED (eligible nodes), CURRENT, READY, UP-TO-DATE, AVAILABLE and NODE SELECTOR. If DESIRED is lower than your node count, some nodes are filtered out by a selector or taint.

Which nodes get a pod

The DaemonSet controller creates one pod per node, then lets the default scheduler bind each pod to that node by adding a node affinity that targets exactly that node. That has a consequence: all normal scheduling filters still apply. A node is eligible only if:

  1. nodeSelector (or affinity.nodeAffinity) in the pod template matches the node's labels. This is how you run an agent only on nodes with a given label, for example disk=ssd or gpu=true. See node selectors and affinity.
  2. Every taint on the node is tolerated by the pod, for the effects that block scheduling. Control-plane nodes are tainted NoSchedule, and nodes that are not ready or unreachable get automatic NoExecute taints. DaemonSet pods automatically get tolerations for several of these built-in taints (such as not-ready, unreachable, disk-pressure, memory-pressure and unschedulable), so an agent keeps running on a struggling node. Others, like the control-plane taint or your own custom taints, need an explicit toleration. See taints and tolerations.
  3. The node has room: if a node cannot fit the pod's resource requests, the pod stays Pending on that node. Give agents small, honest requests, and consider a high priorityClassName for essential ones (such as system-node-critical in kube-system) so they can preempt ordinary pods.

Because eligibility is re-evaluated continuously, the DaemonSet reacts when you relabel a node or change taints: pods are added to newly eligible nodes and removed from nodes that no longer qualify.

A placement stand-in

The simulation below takes a list of nodes with labels and taints, a DaemonSet with a nodeSelector and tolerations, and runs the reconcile logic: create a pod on each eligible node that has none, delete pods from nodes that are no longer eligible. It is a simplified model (it only understands equality selectors and key-only tolerations, and it ignores resource fit), but the shape of the logic matches the real controller.

Python One pod per eligible node

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last two steps: nobody told the DaemonSet to touch worker-1 or worker-2. It noticed the label change and moved the pods on its own. The gpu-1 node never appears for ssd-check even though it has disk=ssd, because of its untolerated gpu taint.

Updating a DaemonSet

spec.updateStrategy has two types:

  • RollingUpdate (default): after you change the pod template, old pods are deleted and replaced node by node. maxUnavailable (default 1) is how many nodes may be without a working agent at once; it can be a number or a percentage. Newer clusters also support maxSurge for DaemonSets, which starts the new pod on a node before killing the old one, so there is no gap in coverage. You cannot use a nonzero maxSurge together with a nonzero maxUnavailable, and hostPort-based agents cannot surge, because two pods cannot bind the same port on one node.
  • OnDelete: pods are replaced only when you delete them yourself. Choose this for components where a surprise restart on every node is too risky, such as a CNI plugin, and you want to upgrade nodes one at a time during maintenance windows.

minReadySeconds works as for Deployments: a new pod must stay ready for that long before it counts as available and the update moves on. DaemonSets keep revision history through ControllerRevision objects (revisionHistoryLimit, default 10), so kubectl rollout undo daemonset/log-agent and rollout history work like they do for Deployments.

Python DaemonSet rolling update in waves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The real controller does not work in synchronized waves: it keeps the number of unavailable nodes at or below the limit and starts the next replacement as soon as the previous one is ready. The wave model shows the limit; it exaggerates the lock-step.

Patterns and cautions

  • One per node is not one per cluster. Use a DaemonSet when the work is tied to nodes. If you only need a few highly available copies, use a Deployment.
  • Keep agents light. A DaemonSet's cost is multiplied by your node count: 100m CPU on 500 nodes is 50 cores. Set requests and limits, and watch the memory use of log agents during traffic bursts (see resource requests and limits).
  • Do not write to the application's namespace. Node-level agents typically live in kube-system or a dedicated monitoring namespace.
  • Agents see node data. Mounting /var/log read-only is far safer than running privileged. Prefer narrow mounts and drop capabilities.
  • Static pods are a related, node-local mechanism: the kubelet itself starts them from files on disk (the control plane often runs this way). A DaemonSet is managed through the API and is the normal choice for add-ons.
  • Alternatives for "run something on a node" exist for one-off tasks, such as kubectl debug node/<name>, which starts a temporary pod on a specific node.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How many pods does a DaemonSet run?

    A DaemonSet has no replicas field. It creates one pod on each node that passes the node selector, affinity and taint checks.
  2. 2. Your DaemonSet has no pods on the control-plane node. What is the most likely cause?

    Control-plane nodes are tainted. Add a toleration for node-role.kubernetes.io/control-plane to the pod template to include them.
  3. 3. What happens to DaemonSet pods when you relabel a node so it no longer matches nodeSelector?

    Eligibility is re-evaluated continuously, so the controller removes pods from nodes that stop qualifying. It does not move them, because there is exactly one per node.
  4. 4. Which update strategy should you use to upgrade a risky node-level plugin one node at a time under manual control?

    With OnDelete the controller replaces a pod only after you delete it, giving you full control of the order and timing.
  5. 5. Why does a small increase in a DaemonSet pod's CPU request matter?

    Every node runs a copy, so 100m on 500 nodes reserves 50 CPU cores in total.

Summary

  • A DaemonSet runs one pod on every eligible node, adding and removing pods as nodes join, leave or change labels and taints.
  • It has no replicas; the count is the number of eligible nodes.
  • Eligibility follows nodeSelector/affinity, taints and tolerations, and resource fit; built-in tolerations keep agents running on unhealthy nodes.
  • Updates use RollingUpdate (with maxUnavailable, and optionally maxSurge) or OnDelete; revisions allow rollback.
  • Use DaemonSets for log, metrics, networking and storage agents, and keep them small because every node pays for them.

Next: Jobs and CronJobs.