Skip to content
Tutorials Center
My learning

Topics

Lesson 71 of 78 ยท Projects

Project: A Tiny Operator

Design a Database custom resource, write its reconcile logic with status conditions and a finalizer, and run a Python operator simulation to see it work.

An operator is a controller that manages an application on your behalf, using a custom resource as its interface. Instead of asking users to assemble a StatefulSet, a Service, a backup bucket and a config file for every database, you let them write five lines of YAML ("I want a Postgres with 3 replicas") and the operator does the rest, forever, including the clean-up when they delete it.

In this project you design a Database custom resource, write the reconcile logic, report progress with status conditions and add a finalizer so that external resources are removed properly. A real operator is a program written with a framework such as kubebuilder, and it needs a cluster, so the YAML and Go below are read-only. The runnable part is a Python simulation of the same logic. It is a stand-in that models the control flow, not the API server or the framework. Background: Controllers and the Reconcile Loop and Custom Resources and Operators.

What an operator is made of

CRD Teaches the API a new kind.
Custom resource One user request.
Controller Makes reality match it.

CRD plus controller equals operator.

1 / 2

Design the custom resource

Good CRD design starts from the user's view. A database user cares about the engine, the version, the number of replicas and the disk size. They do not care about StatefulSet names or headless Services. Keep spec small and declarative, and keep everything the operator learns in status.

YAML
apiVersion: db.example.com/v1alpha1
kind: Database
metadata:
  name: orders-db
  namespace: shop
spec:
  engine: postgres
  version: "17"
  replicas: 3
  storageGB: 20

The matching CustomResourceDefinition registers the kind and describes the allowed shape with an OpenAPI schema. The API server then rejects invalid objects before the operator ever sees them.

YAML
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.db.example.com
spec:
  group: db.example.com
  scope: Namespaced
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames: [db]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Ready
          type: string
          jsonPath: .status.readyReplicas
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, version]
              properties:
                engine:
                  type: string
                  enum: [postgres, mysql]
                version:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 7
                  default: 1
                storageGB:
                  type: integer
                  minimum: 1
                  default: 10
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true

Two details matter. subresources.status makes status a separate endpoint, so users editing spec cannot accidentally overwrite what the operator reports, and the operator's status updates do not count as spec changes. And default values are filled in by the API server, which keeps the controller code free of "if replicas is zero" guesswork. Our first stand-in applies the same rules: defaults, required fields, enum and range checks.

Python Schema validation and defaults for a Database

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reconcile function

The heart of the operator is a function that receives only the name of an object and must work out the rest by looking at the world. It has to be idempotent: running it ten times in a row must give the same result as running it once, because the framework will call it after every relevant event and also after restarts. A typical skeleton in Go with controller-runtime looks like this:

Go
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var db dbv1.Database
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 1. Being deleted? Run the finalizer logic, then let go.
	if !db.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&db, finalizerName) {
			if err := r.deleteExternalBackups(ctx, &db); err != nil {
				return ctrl.Result{}, err // retried with backoff
			}
			controllerutil.RemoveFinalizer(&db, finalizerName)
			return ctrl.Result{}, r.Update(ctx, &db)
		}
		return ctrl.Result{}, nil
	}

	// 2. Make sure our finalizer is present before creating anything external.
	if controllerutil.AddFinalizer(&db, finalizerName) {
		return ctrl.Result{}, r.Update(ctx, &db)
	}

	// 3. Create or update the StatefulSet that this Database owns.
	sts := desiredStatefulSet(&db)
	if err := controllerutil.SetControllerReference(&db, sts, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}
	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
		sts.Spec = desiredStatefulSet(&db).Spec
		return nil
	}); err != nil {
		return ctrl.Result{}, err
	}

	// 4. Report what we observed.
	db.Status.ReadyReplicas = sts.Status.ReadyReplicas
	setCondition(&db, "Available", sts.Status.ReadyReplicas == *sts.Spec.Replicas)
	return ctrl.Result{}, r.Status().Update(ctx, &db)
}

This is a skeleton: helpers such as desiredStatefulSet and setCondition are yours to write, and a real operator would also watch the StatefulSet (with Owns(&appsv1.StatefulSet{})) so that status changes trigger a new pass. Notice the order. The finalizer is added before anything external is created, so there is no window in which something exists that nobody will clean up. Returning an error makes the framework retry with backoff; returning no error and no requeue means "done until something changes".

Finalizers and owner references

Kubernetes has two clean-up mechanisms, and an operator needs both.

  • Owner references link a child to its parent. When the parent Database is deleted, the garbage collector deletes the StatefulSet automatically. Use SetControllerReference for everything inside the cluster.
  • Finalizers are strings in metadata.finalizers. While any finalizer is present, deleting the object only sets deletionTimestamp; the object stays visible until the controller finishes its clean-up and removes its finalizer. Use this for things outside the cluster that the garbage collector cannot see, such as a cloud snapshot bucket or a DNS record.

Without a finalizer, a kubectl delete database orders-db would remove the object at once, and the operator would only find out afterwards, with no way to know what to clean up. A finalizer that is never removed because the operator crashed or has a bug leaves the object stuck in Terminating, which is the most common operator support question. The simulation shows the whole life cycle.

Python A tiny operator: create, scale, delete with a finalizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output from top to bottom. The first pass only adds the finalizer. Later passes create the StatefulSet and report Available=False until every pod is ready. The spec edit simply starts another pass with the same code. On deletion, the object lingers as Terminating until the operator removes the backup bucket and its finalizer; only then does the object disappear and the garbage collector remove the StatefulSet. Nothing is left over.

Status conditions

status.conditions is a convention across Kubernetes: a list of entries with a type (such as Available, Progressing, Degraded), a status of "True", "False" or "Unknown", a machine-readable reason in CamelCase and a human-readable message. Tools such as kubectl wait --for=condition=Available database/orders-db rely on it. Keep conditions about the current state, not history, and only update lastTransitionTime when the status actually flips. The next stand-in shows how an operator can derive conditions from observed facts instead of tracking them as flags.

Python Deriving conditions from observations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Taking it further

To turn this into a real operator, scaffold it with kubebuilder (kubebuilder init, kubebuilder create api), generate the CRD from Go types, run the controller locally against a kind cluster, and write tests with envtest. Add leader election so that only one replica of the operator reconciles at a time, RBAC rules generated from markers, metrics and events (r.Recorder.Event) for debugging. Consider whether you need an operator at all: a Helm chart (Project: Package an App as a Helm Chart) is simpler when you only need templating, and an operator pays off when the application needs ongoing, stateful operations such as failover, backups and upgrades.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must a reconcile function be idempotent?

    Controllers are triggered by events and periodic resyncs, and retries happen after errors. Running the function repeatedly must converge to the same result without duplicating work.
  2. 2. What does a finalizer on a custom resource do when you run kubectl delete?

    With finalizers present, the API server only sets deletionTimestamp. The object disappears after every finalizer has been removed by the controller that owns it.
  3. 3. Which clean-up mechanism fits an object that lives outside the cluster, such as a cloud snapshot?

    The garbage collector only follows owner references between Kubernetes objects. External resources need a finalizer so the operator can delete them before the object goes away.
  4. 4. Why enable the status subresource in the CRD?

    With the subresource, /status has its own endpoint and permissions. Editing spec does not touch status, and status updates do not bump the object's generation.

Summary

  • An operator is a CRD plus a controller; design spec from the user's point of view and keep observations in status.
  • Reconcile receives a name, reads the current world, makes the smallest fix and must be idempotent.
  • Owner references clean up children inside the cluster; finalizers clean up external resources before an object disappears.
  • Add the finalizer before creating anything external, and always make sure it can be removed.
  • Status conditions give tools and people a standard way to see whether the resource is available, progressing or degraded.
  • The Python simulation models the control flow; the real thing is a Go (or other language) program running in the cluster.

Next, combine several controllers in one program: Project: Build a Mini Cluster Simulator.