Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 78 ยท Extending Kubernetes

Custom Resources and Operators

Extend Kubernetes with CustomResourceDefinitions and operators: schemas, validation, kubebuilder and operator-sdk, plus a runnable Database operator stand-in.

Kubernetes ships with types such as Pod, Deployment and Service. But your platform has its own concepts: a database, a certificate, a backup schedule, a tenant. You could describe them with a Deployment, a Service, a Secret and a CronJob stitched together, and hope everyone remembers how. A better way is to teach Kubernetes the new word. A CustomResourceDefinition (CRD) adds a new kind to the API, and a controller gives it meaning. Together they are called an operator: operational knowledge, written as code, running inside the cluster.

After this lesson you will be able to read any CRD and operator you meet, such as cert-manager or the Prometheus Operator, and know how to start building your own. It builds directly on Controllers and the Reconcile Loop and the API groups and versions lesson.

You cannot register a CRD or run an operator inside this page. The YAML and Go below are read-only. The Python programs are stand-ins that validate a custom resource against a schema and run an operator's reconcile logic against a fake cluster; they show the behavior, not the real API machinery.

Operator = CRD + controller

CRD Adds a new kind and its schema to the API.
Custom resource An instance, like a Database named shop-db.
Controller Reconciles the resource into real objects.

The API server stores custom resources like any other object.

1 / 2

Defining a new kind

A CRD is itself a Kubernetes object. This one introduces a Database kind in the group db.example.com:

YAML
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.db.example.com      # must be <plural>.<group>
spec:
  group: db.example.com
  scope: Namespaced
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames: [db]
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Ready
          type: string
          jsonPath: .status.phase
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, storageGB]
              properties:
                engine:
                  type: string
                  enum: [postgres, mysql]
                version:
                  type: string
                  default: "16"
                storageGB:
                  type: integer
                  minimum: 1
                  maximum: 1000
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 5
                  default: 1
              x-kubernetes-validations:
                - rule: "self.storageGB >= oldSelf.storageGB"
                  message: "storage can grow but never shrink"
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true

After kubectl apply -f database-crd.yaml the API server serves new REST paths under /apis/db.example.com/v1/ and kubectl get databases (or kubectl get db) works immediately. Notes on the parts:

  • scope is Namespaced or Cluster, as for built-in kinds.
  • versions can list several versions; exactly one has storage: true and is what etcd stores. Moving between versions needs a conversion strategy (None just changes the apiVersion string, or a conversion webhook for real changes).
  • schema.openAPIV3Schema is required in v1 CRDs. It validates every create and update, applies defaults, and prunes unknown fields: a typo like replcas is silently dropped instead of stored, a surprise worth remembering.
  • x-kubernetes-validations holds rules written in CEL (the Common Expression Language). oldSelf makes a rule a transition rule, comparing the new value with the stored one, here to forbid shrinking storage.
  • subresources.status gives the object a separate /status endpoint. Users write spec, the controller writes status, and neither can accidentally overwrite the other.

An instance, a custom resource, then looks like any manifest:

YAML
apiVersion: db.example.com/v1
kind: Database
metadata:
  name: shop-db
spec:
  engine: postgres
  version: "16"
  storageGB: 20
  replicas: 2

Note the quoted "16"; an unquoted 16 would be an integer and fail the string schema (see YAML and the Shape of a Manifest). The next stand-in is a mini validator with the same rules: types, required fields, enum, bounds, defaults and pruning.

Python Validating a custom resource against its schema

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What an operator does

The CRD alone only stores data. The operator is a controller that watches Database objects and creates whatever is needed: a StatefulSet for the pods, a Service for clients, a Secret with credentials, a CronJob for backups. It also reacts to changes (scale up, bump the version, grow storage), repairs damage, and reports back in status. The user's side shrinks to ten lines of YAML; the knowledge lives in the operator. This is why complex stateful software (databases, message brokers, certificate management) often ships as an operator.

Good operators follow the same rules as any controller. They set owner references on the child objects so deleting the Database removes everything it created; they are idempotent; they write status.conditions (such as Ready=True, with a reason and message) so users and tools can see health at a glance; and they use finalizers when something outside the cluster, like a cloud volume or a final backup, must be cleaned up before the object may disappear.

The stand-in below is an operator for the Database kind. A fake StatefulSet controller brings up one replica per pass, as the real one does for ordered pods, and the operator turns spec changes into child objects and status.

Python A Database operator reconciling a custom resource

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the second edit: the operator updated only the fields that changed and left the ready counter alone. The third edit shows the operator reporting a problem through status instead of crashing, so a later correction fixes it. In a real operator, the CRD schema would already have rejected oracle when you applied the change; validation in code is a second line of defense for rules the schema cannot express.

Building one: kubebuilder and operator-sdk

You rarely write the plumbing by hand. kubebuilder is the standard scaffolding tool for Go operators on top of the controller-runtime library, and operator-sdk builds on it (adding extras such as bundles for OLM, the Operator Lifecycle Manager, and operators driven by Helm charts or Ansible). Other languages work too: Kopf for Python, Java Operator SDK, Kube-rs for Rust. A typical start:

Terminal
kubebuilder init --domain example.com --repo example.com/db-operator
kubebuilder create api --group db --version v1 --kind Database --resource --controller
make manifests   # generate the CRD YAML from Go types and markers
make install     # install CRDs into the current cluster
make run         # run the controller locally against that cluster

You describe the API as Go structs; special comments (markers) become schema rules:

Go
type DatabaseSpec struct {
	// +kubebuilder:validation:Enum=postgres;mysql
	Engine string `json:"engine"`
	// +kubebuilder:validation:Minimum=1
	StorageGB int32 `json:"storageGB"`
	// +kubebuilder:default=1
	Replicas int32 `json:"replicas,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
type Database struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              DatabaseSpec   `json:"spec,omitempty"`
	Status            DatabaseStatus `json:"status,omitempty"`
}

The heart of the controller is one function. It receives only a name, exactly like the work queue keys in the previous lesson:

Go
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var db dbv1.Database
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	sts := &appsv1.StatefulSet{ObjectMeta: metav1.ObjectMeta{Name: db.Name, Namespace: db.Namespace}}
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
		sts.Spec.Replicas = &db.Spec.Replicas
		// ... fill in the pod template from db.Spec ...
		return controllerutil.SetControllerReference(&db, sts, r.Scheme)
	})
	if err != nil {
		return ctrl.Result{}, err // the queue retries with backoff
	}
	// ... update db.Status ...
	return ctrl.Result{}, nil
}

func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&dbv1.Database{}).
		Owns(&appsv1.StatefulSet{}).
		Complete(r)
}

For says which kind to reconcile and Owns says that changes to owned StatefulSets should enqueue their parent Database, the same trick as pods enqueuing their ReplicaSet. CreateOrUpdate makes the function idempotent.

Operators you will meet

  • cert-manager issues and renews TLS certificates (Certificate, Issuer); see Ingress.
  • Prometheus Operator manages Prometheus and defines ServiceMonitor and PrometheusRule; see Metrics and Monitoring.
  • Database operators such as CloudNativePG and the MySQL and MongoDB operators handle failover and backups; see Running Databases and Backups.
  • Argo CD and Flux are operators too: their Application and Kustomization kinds are CRDs (GitOps).
  • KubeVirt adds a VirtualMachine kind, so full virtual machines are declared, scheduled and managed with the same API and tools as containers. It is used when a legacy workload cannot be containerized yet but should live next to the Pods in one cluster.
  • Strimzi (Kafka), Crossplane (cloud resources as Kubernetes objects) and Gateway API implementations all follow this pattern.

Before you write an operator, check whether one exists and whether you need one: a Helm chart or a few manifests are often enough. Operators earn their cost for ongoing operational logic, not just initial setup. Each CRD you install also becomes API surface you must secure with RBAC and keep upgraded.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is an operator?

    An operator pairs a custom API type with a controller that holds operational knowledge and turns custom resources into real, managed objects.
  2. 2. A custom resource contains a misspelled field that is not in the CRD schema. What usually happens?

    Structural schemas prune unknown fields by default. That is why typos in custom resource specs can go unnoticed without validation tooling.
  3. 3. Why enable the status subresource on a CRD?

    With the subresource, status is updated through its own endpoint. Normal updates ignore status changes and status updates ignore spec changes.
  4. 4. In controller-runtime, what does Owns(&appsv1.StatefulSet{}) do?

    Owns registers a watch on the child type and maps each event to the owner via the owner reference, so damage to a child triggers a reconcile of the parent.
  5. 5. Which names a CRD correctly?

    The CRD object's name must be the plural resource name followed by the API group.

Summary

  • A CRD adds a new kind (with schema, defaults, validation rules and an optional status subresource) to the Kubernetes API; instances are called custom resources.
  • An operator is a controller for such a kind: it watches the resources, creates and updates child objects with owner references, and reports health in status.conditions.
  • Schemas prune unknown fields and validate types, enums and bounds; CEL rules in x-kubernetes-validations can express more, including transition rules with oldSelf.
  • kubebuilder and operator-sdk generate the scaffolding, CRD YAML and controller-runtime wiring for Go operators.
  • Reach for an operator when ongoing operational logic needs automating; use a simpler tool when installation alone is the problem.

Next, look at what your workloads say about themselves: Logging in Kubernetes.