Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 39 ยท Operations

Running Kafka on Kubernetes

Run Kafka on Kubernetes with the Strimzi operator: custom resources, StatefulSets, persistent storage, rack awareness, listeners and rolling updates, with runnable stand-ins.

Many companies run everything on Kubernetes, so they want Kafka there too. It can work well, but Kafka is a stateful system: each broker owns data on disk, has a stable identity, and must be restarted carefully. Kubernetes was designed first for stateless services, so you need the right tools to bridge that gap.

This lesson explains what Kubernetes features matter for Kafka, why people use an operator such as Strimzi, and how storage, rack awareness and listeners map onto it.

Real versus stand-in. The YAML below follows the Strimzi operator's custom resources and is read-only here, because the site cannot run Kubernetes or Kafka. Check the Strimzi documentation for the exact apiVersion and field names of the version you install. The TryIt blocks are plain JavaScript stand-ins for placement and rolling restarts, not Kubernetes.

What makes Kafka different from a web app

A web service pod is interchangeable: kill it, start another, nobody notices. A Kafka broker is not:

  • Identity: other brokers and clients address it by a fixed broker id and a stable DNS name.
  • Data: its partitions live on its disk. A replacement must get the same volume back.
  • Order of operations: restarting two brokers at once can drop partitions below min.insync.replicas and stop writes (see replication and ISR).
  • Network: clients outside the cluster must reach each broker individually, not through a load balancer that picks any pod.

Kubernetes answers the first two with a StatefulSet (stable names such as kafka-0, kafka-1, and per-pod persistent volume claims) and a headless Service that gives each pod its own DNS record. The last two need extra care.

Why an operator

You could write StatefulSets, ConfigMaps, Services and certificates by hand. People rarely do. An operator is a controller that watches custom resources and does the operational work a human would: creating brokers, generating TLS certificates, rolling them safely, scaling, and managing users and topics.

Strimzi is the most widely used open-source operator for Apache Kafka (a CNCF project). Other options include vendor operators and managed services. With Strimzi you describe the cluster declaratively:

YAML
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: demo
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    listeners:
      - name: internal
        port: 9092
        type: internal
        tls: false
      - name: secure
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
    rack:
      topologyKey: topology.kubernetes.io/zone
  entityOperator:
    topicOperator: {}
    userOperator: {}

Brokers (and KRaft controllers) are declared as node pools, which let you give different roles or storage to different groups of nodes:

YAML
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: brokers
  labels:
    strimzi.io/cluster: demo
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: fast-ssd
        deleteClaim: false

Older Strimzi versions with ZooKeeper put replicas and storage directly into spec.kafka. The cluster layout concepts are those of Cluster Architecture and KRaft.

Topics and users become Kubernetes objects too, so they can live in Git next to the application:

YAML
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  labels:
    strimzi.io/cluster: demo
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 604800000

Storage

Kafka is disk-bound, so storage decisions matter most.

  • Use persistent volumes, not emptyDir. When a pod is rescheduled, the claim follows it, and the broker keeps its log and only has to catch up on what it missed.
  • Prefer fast block storage (SSD-backed volumes) over network file systems such as NFS, which Kafka's file-handling does not suit.
  • Local disks are fastest but tie the broker to one machine. If the node is lost, the broker must rebuild its data by replicating from other brokers, which is fine only if the replication factor is 3 and the cluster can afford the traffic.
  • deleteClaim: false keeps the volume even if you delete the Kafka resource. This prevents an accidental kubectl delete from erasing the data.
  • Size disks for retention plus headroom; resizing a volume claim needs a storage class that allows expansion.

Rack awareness across zones

Kafka's rack awareness spreads replicas of a partition across racks (or cloud availability zones), so losing a zone does not lose all copies. A broker advertises its rack (broker.rack); the placement logic prefers different racks for the replicas of one partition. Strimzi fills broker.rack from the node label named in rack.topologyKey, and also supports consumers fetching from the closest replica.

Pair this with Kubernetes pod anti-affinity or topology spread constraints so brokers are not all scheduled on the same machine in the first place.

JS Rack-aware replica placement

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Listeners: reaching the brokers

Inside the cluster, clients use the headless service DNS names or the bootstrap service, which is simple. The hard part is external access. A Kafka client contacts the bootstrap address, receives the metadata containing each broker's advertised address, and then connects to the leader of each partition directly. So every broker needs an address reachable from outside, not only a single entry point.

Strimzi listener types cover the usual choices:

YAML
listeners:
  - name: external
    port: 9094
    type: loadbalancer   # one LoadBalancer per broker plus a bootstrap one
    tls: true
    authentication:
      type: scram-sha-512

Other types are nodeport, ingress (TLS passthrough only, since Kafka is not HTTP) and route on OpenShift. Security settings follow Security and Quotas. The operator generates certificates and per-broker advertised addresses for you, which is a large part of its value.

The simplest rule: keep clients inside the same Kubernetes cluster when you can, and add a carefully planned external listener only for what must come from outside.

Rolling updates and disruption

When you change a broker config or upgrade, the operator restarts brokers one at a time. Before restarting a broker it checks that the restart will not push any partition below min.insync.replicas, and it waits until the restarted broker has rejoined the ISR before moving on.

You can protect against Kubernetes' own disruptions (node drains, cluster upgrades) with a PodDisruptionBudget, which Strimzi creates, limiting how many brokers can be voluntarily evicted at once.

JS A safe rolling restart

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Practical advice

  • Resources: set CPU and memory requests equal to limits so pods get a guaranteed slice. Leave plenty of RAM outside the JVM heap, because Kafka relies on the operating system's page cache.
  • Monitoring: Strimzi can expose JMX metrics for Prometheus; track the lag and under-replicated metrics from Monitoring and Lag.
  • Separate failure domains: run Kafka on its own node pool, so noisy neighbours do not steal disk and network.
  • Backups and DR: Kubernetes does not back up your topics. Use replication across clusters (MirrorMaker 2) for disaster recovery.
  • Consider managed Kafka if you do not want to operate this at all. Running Kafka well on Kubernetes still requires Kafka knowledge.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a StatefulSet used for Kafka brokers rather than a Deployment?

    Each broker has a fixed id, a stable network name and a log on disk that must follow it when rescheduled.
  2. 2. What does an operator such as Strimzi add over hand-written manifests?

    The operator encodes the operational knowledge: safe restarts, certificate handling, scaling and declarative topics and users.
  3. 3. Why is external access to Kafka harder than to a web service?

    After bootstrap, the client uses the advertised addresses from metadata, so a single load balancer that picks any pod does not work.
  4. 4. What is rack awareness for?

    With broker.rack set from the zone label, replicas of one partition are placed in different zones.

Summary

  • Kafka brokers are stateful: they need stable identity, persistent storage and careful, one-at-a-time restarts.
  • Kubernetes provides StatefulSets, volumes and headless Services; an operator like Strimzi adds the Kafka-specific automation.
  • Declare clusters, node pools, topics and users as custom resources and keep them in Git.
  • Use fast persistent block storage and keep volume claims on deletion.
  • Rack awareness plus topology spread keeps replicas in different zones.
  • External access needs per-broker advertised addresses; prefer in-cluster clients where possible.

Next: copy data between clusters in MirrorMaker 2.