Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Operations

Running a Cluster: KRaft, Brokers and Controllers

Understand how a Kafka cluster runs without ZooKeeper: KRaft controllers, broker roles, listeners, rack awareness, sizing and rolling upgrades, with a runnable election stand-in.

Until now we treated the cluster as a set of brokers that somehow agree on who leads each partition. This lesson opens that box. Someone has to keep the cluster's metadata: which topics exist, how many partitions they have, which broker leads each one, which replicas are in sync. Older Kafka versions stored this in Apache ZooKeeper, a separate system you had to run and secure. Modern Kafka keeps it inside Kafka itself, using a protocol called KRaft (Kafka Raft). ZooKeeper mode was removed completely in Kafka 4.0, so new clusters are KRaft clusters.

You will learn the two node roles, how listeners and advertised addresses work (the source of many "I cannot connect" questions), how to spread replicas across racks, how to think about sizing, and how a rolling upgrade goes.

Real versus stand-in. The configuration and shell commands below are real Kafka usage, shown read-only because the site cannot start brokers. The TryIt blocks are hand-written JavaScript stand-ins: a simplified Raft-style election and a capacity calculator. They copy the idea (terms, majority votes, replicated metadata), not the real implementation.

Two roles, one process

Broker Stores partitions and serves clients.
Controller Keeps the cluster metadata.
Combined One process with both roles. Fine for development.

Production clusters usually run dedicated controllers.

1 / 2

The metadata log

KRaft stores cluster metadata as a log, which is exactly the idea from The Commit Log Idea. A small set of controller nodes forms a quorum. One of them is the active controller, the Raft leader; the others are followers that replicate its metadata log, stored in an internal topic called __cluster_metadata.

Every change, such as "create topic orders with 6 partitions" or "broker 3 left the in-sync replica set of orders-2", becomes a record in that log. A record counts as committed once a majority of the quorum has it. Brokers fetch the metadata log too and keep the current state in memory, so they do not ask a database on every request.

The quorum size decides how many failures you survive:

Controllers Majority Failures tolerated
1 1 0 (development only)
3 2 1
5 3 2

Use an odd number. Four controllers tolerate no more failures than three, because the majority of four is three. Three controllers suit most clusters; five are for deployments that must survive two simultaneous losses, for example during maintenance.

When the active controller dies, the followers hold an election. Time is divided into numbered terms. A follower that stops hearing from the leader becomes a candidate, increments the term, votes for itself and asks the others for votes. Each node votes at most once per term, and only for a candidate whose log is at least as up to date as its own. A candidate that collects a majority becomes the leader. This is why a quorum cannot split into two leaders in the same term.

A controller election you can run

The stand-in models three controllers. Each has a term, a vote, and the length of its metadata log. A candidate wins with two of three votes. Then the leader dies and another node takes over, and finally a second failure leaves a lone survivor that can neither win an election nor commit a write.

JS Raft-style controller election

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last steps matter. Node 1 returns holding an older log, so it loses the vote it asks for: nobody votes for a candidate whose log is shorter than theirs. Node 2 wins instead, with node 1's help, and the write that failed earlier now commits. In real KRaft the returning node also truncates or catches up its log from the leader. The majority rule is the same one that protects partitions in Replication, Leaders and ISR.

Configuring a node

A KRaft node is configured by a server.properties file. This is a minimal combined node for local development, where one process is both broker and controller:

PROPERTIES
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

log.dirs=/var/lib/kafka/data

The key settings:

  • process.roles is broker, controller or both. Production clusters normally run separate controller nodes (process.roles=controller) and broker nodes (process.roles=broker), so a busy broker cannot starve the metadata quorum.
  • node.id is a number unique in the cluster. It replaces the old broker.id.
  • controller.quorum.voters lists the static quorum as id@host:port, for example 1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093. Newer Kafka versions can also grow and shrink the quorum dynamically; check the documentation for your version.
  • controller.listener.names names the listener the controllers use among themselves.

Before the first start, the storage directory has to be formatted with a cluster id that every node shares:

Terminal
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/server.properties
bin/kafka-server-start.sh config/server.properties

You generate the id once and reuse it on every node. A node refuses to start on a directory formatted for a different cluster, which protects you from joining the wrong cluster by mistake.

Listeners and advertised listeners

Most connection problems come from mixing up two settings.

  • listeners is what the broker binds to: interface, port and protocol.
  • advertised.listeners is the address the broker tells clients to use after they connect.

Remember how clients work: bootstrap.servers only gets the client to one broker. That broker returns the cluster metadata, including the advertised address of every partition leader, and the client then connects to those addresses. If a broker in a container advertises localhost:9092, a client on another machine will find itself talking to its own localhost and fail, even though the first connection worked.

The usual fix is separate listeners for separate audiences:

PROPERTIES
listeners=INTERNAL://:9092,EXTERNAL://:9094,CONTROLLER://:9093
advertised.listeners=INTERNAL://broker1.internal:9092,EXTERNAL://kafka.example.com:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SASL_SSL,CONTROLLER:PLAINTEXT
inter.broker.listener.name=INTERNAL

Brokers talk to each other over INTERNAL; outside clients use EXTERNAL, with encryption and authentication (see Security and Quotas). Controller nodes need only the CONTROLLER listener.

Rack awareness

If all three replicas of a partition live in the same rack or availability zone, one outage takes the partition away. Set broker.rack on each broker, for example the zone name:

PROPERTIES
broker.rack=eu-central-1a

When Kafka places replicas for a new topic, it spreads them across racks where it can. Consumers can benefit too: with client.rack set on the consumer and replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector on the brokers, a consumer may fetch from a follower in its own zone instead of the remote leader, which saves cross-zone traffic. Producers still write only to leaders.

Sizing: partitions, disks and memory

Capacity planning is arithmetic plus margin. The disk needed for a topic is roughly

incoming bytes per second x retention seconds x replication factor,

split across the brokers, plus headroom (a common habit is to plan for 30 to 40 percent free space, because running out of disk is a severe failure). Network matters as much as disk: every byte produced is read by followers, and consumers multiply the outgoing traffic.

Partitions are not free. Each one has files on disk, open handles, replication work, and metadata that controllers and brokers keep in memory. More partitions also mean longer leader elections after a broker failure and more consumers' worth of coordination. As a conservative starting point, keep partitions per broker in the low thousands and measure before going far beyond that. KRaft raised the cluster-wide ceiling compared with ZooKeeper, but it did not make partitions cost nothing. Use the throughput reasoning from Topics and Partitions: enough partitions for your consumer parallelism and growth, not a huge number "just in case".

The calculator below turns the formula into numbers.

JS Disk and partition sizing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the second result carefully: a modest-looking 200 MB/s with a week of retention needs hundreds of terabytes across the cluster (about 346 TB with replication in this example). Retention is usually the biggest sizing lever (see Retention, Segments and Disk Layout).

Rolling upgrades

You upgrade a Kafka cluster without downtime by restarting one node at a time. The general outline for a KRaft cluster is:

  1. Read the release notes for your target version, especially the upgrade section and removed settings.
  2. Replace the binaries and restart the controllers one by one, waiting for each to rejoin the quorum.
  3. Restart the brokers one by one. Before stopping a broker, check that there are no under-replicated partitions, and wait for it to catch up again before moving on (see Monitoring, Lag and Troubleshooting).
  4. When every node runs the new software, raise the cluster's finalized feature level, which enables the new metadata format. The tool is kafka-features.sh:
Terminal
bin/kafka-features.sh --bootstrap-server localhost:9092 describe
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --metadata 4.0

Keeping the feature level unchanged until the end is what makes a rollback possible during the first steps. Controlled shutdown, which is the default behaviour when you stop a broker normally, moves partition leadership away before it exits, so clients see only a short pause.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does KRaft replace?

    KRaft keeps cluster metadata in a replicated log managed by Kafka's own controllers, so no separate ZooKeeper cluster is needed. ZooKeeper mode was removed in Kafka 4.0.
  2. 2. A cluster has five controller nodes. How many can fail while the quorum still works?

    The majority of five is three, so two nodes may fail. A quorum needs a majority of all voters, not just of the survivors.
  3. 3. A client connects to bootstrap.servers successfully but then times out talking to the partition leaders. What is the most likely cause?

    The bootstrap server returns the advertised addresses of the leaders. If those are wrong for the client's network, only the first connection works.
  4. 4. Why is an even number of controllers a poor choice?

    Four controllers need three votes for a majority, so they tolerate one failure, just like three controllers. The extra node costs money without adding resilience.

Summary

  • KRaft stores cluster metadata in a replicated log on a controller quorum; ZooKeeper is gone as of Kafka 4.0.
  • A node has the role broker, controller or both; production usually separates them. Run 3 or 5 controllers.
  • Elections use terms and majority votes, and only a candidate with an up-to-date log can win.
  • listeners is what a broker binds to, advertised.listeners is what clients are told to use. Use separate listeners for internal and external traffic.
  • Set broker.rack to spread replicas across zones.
  • Size disk from throughput x retention x replication factor with headroom, and treat partition count as a cost.
  • Upgrade controllers first, then brokers one at a time, and finalize the feature level last.

Next, learn what to watch while the cluster runs in Monitoring, Lag and Troubleshooting.