Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 78 ยท Extending Kubernetes

Controllers and the Reconcile Loop

Understand how Kubernetes controllers work: level-triggered logic, informers, work queues, idempotency and owner references, with a runnable controller.

Every behavior you have seen so far is produced by a controller. The Deployment controller creates ReplicaSets, the ReplicaSet controller creates pods, the endpoints controller keeps Services pointed at ready pods, and the node controller marks vanished nodes as unhealthy. Kubernetes is less a single program than a crowd of small, independent control loops that all talk to the same API server. If you understand one controller, you understand the whole system, and you can write your own (the next lesson, Custom Resources and Operators, does exactly that).

This lesson looks inside a controller: how it learns about changes, why it works on keys instead of events, how a work queue deals with bursts and failures, and how owner references let Kubernetes clean up after itself. You met the basic loop in What Is Kubernetes?; here it gets its machinery.

A real controller is written in Go with client-go or controller-runtime and talks to a real API server. The Python programs in this lesson are stand-ins: they reproduce the structure (store, informer-style events, rate-limited queue, reconcile function, garbage collector) in a few dozen lines, but they are not the real libraries.

Level-triggered, not edge-triggered

Imagine a controller that reacted to events like "pod deleted" by creating a pod. If it crashed for a minute and missed the event, the pod would never come back. This is edge-triggered logic: it depends on seeing every change.

Kubernetes controllers are level-triggered: an event is only a hint that "something about this object changed, take a look". The controller then reads the current desired state and the current actual state and acts on the difference. It does not matter whether it saw one event or fifty, or none at all and merely woke up on a timer. The result is the same, which makes controllers robust against restarts, duplicates, reordering and lost notifications.

Rules of a good reconciler

Level-based Compare states, ignore the event history.
Idempotent Running it twice changes nothing the second time.
Stateless Read everything fresh from the API each time.
Retry on error Return an error and the key is tried again.

These four rules explain most controller bugs.

1 / 2

Informers: watching without hammering

A controller must know when objects change, but asking the API server "what is new?" in a loop would overload it. Instead the API supports watch: a long-lived HTTP stream that delivers ADDED, MODIFIED and DELETED events. An informer wraps this. It first lists all objects of a type and stores them in a local in-memory cache, then watches from the list's resourceVersion onwards, updating the cache as events arrive. If the stream breaks, it lists again and carries on.

Two consequences matter in practice. Reads inside reconcile come from the cache, not from the API server, so they are fast and cheap (and may be a few milliseconds behind). And a shared informer serves every controller in the process, so ten controllers watching pods open one stream, not ten.

When an event arrives, the informer's handler does not do the work. It computes the key of the object that needs attention, usually namespace/name, and adds it to the work queue. For a pod owned by a ReplicaSet, the key is the ReplicaSet's, because that is the object whose desired state may now be unsatisfied.

flowchart LR
  API["API server"] -->|"list + watch"| Inf["Informer<br/>(local cache)"]
  Inf -->|"event: enqueue key"| Q["Work queue<br/>(deduplicated)"]
  Q -->|"next key"| W["Worker: reconcile(key)"]
  W -->|"reads"| Inf
  W -->|"create / update / delete"| API
  W -.->|"error: retry with backoff"| Q

Events become keys; the worker reads state from the cache and writes changes back to the API.

The work queue

The queue between handlers and workers has three jobs.

  1. Deduplication. If a key is already waiting, adding it again does nothing. A burst of twenty changes to one ReplicaSet leads to one reconcile, which sees the latest state anyway.
  2. No parallel work on one key. The same key is never processed by two workers at once, so a reconcile does not race with itself.
  3. Rate-limited retries. If reconcile returns an error, the key is put back after a delay that grows with each consecutive failure (a typical limiter starts at a few milliseconds and doubles up to several minutes). On success the controller calls Forget, which resets the backoff for that key. This stops a permanently failing object from consuming all the controller's attention.

Here is a controller built from these parts. It manages a ReplicaSet-like object, with an API that fails on the first two pod creations to show the retry.

Python A controller with a work queue and retries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three events produced one queue entry, and the reconcile saw only the final replicas: 4. The two timeouts caused retries after 1 and then 2 ticks, and once the API recovered the controller finished. Deleting a pod by hand needed no special case: the pod's deletion event enqueued its owner and the same function restored the count. The extra reconciles triggered by the controller's own pod creations found nothing to do, which is what idempotency means.

Idempotency and status

A reconciler must be safe to run repeatedly, so write it as "make it so" instead of "do this once". Avoid logic such as "create a pod because an event said so"; prefer "count the pods, and create the difference". When creating child objects, give them deterministic or generated-but-recorded names, and handle "already exists" as success. Never keep important state only in the controller's memory, because it may restart at any moment, and with leader election (a Lease object that one replica holds at a time) another replica may take over.

Controllers also report back. Each object has a status section that the controller updates with what it observed: ready replicas, conditions such as Available, and observedGeneration (the metadata.generation it last acted on, so clients can tell whether status is current). Users write spec; controllers write status.

Owner references and garbage collection

When the Deployment controller creates a ReplicaSet, it stamps the new object with an owner reference pointing back at the Deployment. The same happens from ReplicaSet to pods.

YAML
metadata:
  name: web-7d9f8c6b5
  ownerReferences:
    - apiVersion: apps/v1
      kind: Deployment
      name: web
      uid: 5a1f0c1e-0000-0000-0000-000000000000
      controller: true
      blockOwnerDeletion: true

(The uid is illustrative.) These links do two things. They tell the owning controller which objects are its children (controller: true marks the managing owner, and a ReplicaSet will not adopt a pod that another controller owns), and they drive the garbage collector: when an owner is deleted, the collector removes dependents whose owner no longer exists. The deletion mode is chosen when you delete:

  • Background (the default): the owner disappears immediately and the collector removes the dependents afterwards.
  • Foreground: the owner stays, marked with a deletion timestamp and the foregroundDeletion finalizer, until all dependents with blockOwnerDeletion are gone.
  • Orphan: the owner is deleted and the dependents survive with their owner reference removed (kubectl delete --cascade=orphan).

Owner references only work inside one namespace (or from a namespaced child to a cluster-scoped owner), never from a cluster-scoped object to a namespaced one. Finalizers, which let a controller run clean-up before an object disappears, appear again in Project: A Tiny Operator.

Python Owner references and garbage collection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the order: foreground removes the pods first and the Deployment last, background removes the Deployment first, and orphan leaves the ReplicaSet and its pods running with no parent.

Practical advice

  • Return an error only for conditions that a retry can fix; for permanent problems, record a status condition and stop retrying.
  • Keep each reconcile fast, and let the queue bring you back instead of sleeping inside the function.
  • Watch the objects you create (not just the one you manage), so manual damage triggers a repair.
  • Use kubectl get -w or kubectl get events to watch controllers work, and read the API object model again if the flow feels abstract.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does level-triggered mean for a controller?

    Events are only hints. The controller reads the current state and fixes the difference, so lost or duplicate events do not matter.
  2. 2. Why does the work queue deduplicate keys?

    A burst of changes to the same object needs just one reconcile, since it reads the final state anyway.
  3. 3. A reconcile function returns an error. What normally happens to the key?

    The rate-limited queue retries with exponential backoff, and Forget resets the delay once a reconcile succeeds.
  4. 4. You run kubectl delete deployment web --cascade=orphan. What happens to its ReplicaSet and pods?

    Orphan deletion removes only the owner and clears the owner reference on its dependents, which therefore keep running.
  5. 5. Where does a reconcile function usually read object state from?

    Informers keep an in-memory cache fed by the watch stream, so reads are cheap. Controllers never talk to etcd directly.

Summary

  • Kubernetes behavior comes from many independent controllers that run level-triggered reconcile loops against the API server.
  • Informers list and watch objects, keep a local cache and turn events into keys on a work queue.
  • The queue deduplicates keys, never runs one key twice at once and retries failures with growing backoff.
  • Reconcilers must be idempotent and stateless: read current state, act on the difference, write status.
  • Owner references link children to parents; the garbage collector deletes dependents in background, foreground or orphan mode.

Next, extend the API with your own types: Custom Resources and Operators.