Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 39 ยท Topics and Partitions

Replication, Leaders and ISR

Understand Kafka replication: leaders and followers, the in-sync replica set, acks and min.insync.replicas, the high watermark, and what unclean election risks.

Disks fail and machines crash. A topic that lives on only one broker would lose data and become unavailable the moment that broker disappeared. Kafka solves this with replication: every partition is copied to several brokers, and the cluster keeps working when some of them fail.

Replication is where Kafka's durability promises come from, and also where most of its surprising settings live: acks, min.insync.replicas, the in-sync replica set, the high watermark. This lesson explains how they fit together, using a small simulation of one partition and its replicas.

Leaders and followers

Each partition has a replication factor, the number of copies. A factor of 3 is the usual production choice. One of the replicas is the leader; the others are followers.

  • Producers write only to the leader.
  • Consumers normally read from the leader too. (Newer versions can allow reads from a nearby follower, a feature aimed at saving cross-zone network traffic.)
  • Followers continuously fetch new records from the leader, the same way a consumer would, and append them to their own copy.

Leadership is spread over the cluster: broker 1 may lead partition 0 of orders while broker 2 leads partition 1. So every broker is a leader for some partitions and a follower for others, which spreads the load.

Roles for each partition

Leader Takes all writes and serves reads.
Followers Fetch from the leader and copy the log.
ISR Replicas that are caught up with the leader.

The in-sync replica set (ISR) always includes the leader.

1 / 2

The in-sync replica set

A follower is in sync when it is keeping up with the leader. The leader tracks this, and a follower that stops fetching or falls too far behind for longer than replica.lag.time.max.ms (30 seconds by default) is removed from the ISR. When it catches up again, it rejoins. The ISR is therefore the set of replicas that are guaranteed to hold every committed record.

Only members of the ISR can be elected leader (in the default, safe configuration). That is the key to the durability story: any replica that could take over has all committed data.

The high watermark

The leader knows the log end offset of each ISR member. The smallest of those is the high watermark (HW): all records below it exist on every in-sync replica, so they are committed. Consumers can read only up to the high watermark. A record that is only on the leader is not yet visible, because if the leader crashed now, that record could vanish after a new leader takes over, and consumers would have seen something that never existed.

acks: how safe is "sent"?

The producer's acks setting decides when a write counts as successful.

  • acks=0: the producer does not wait for any answer. Fastest, and records can vanish unnoticed.
  • acks=1: the leader writes the record to its own log and answers. If the leader crashes before followers copy it, the record is lost even though the producer was told it succeeded.
  • acks=all (or -1): the leader answers only after every replica currently in the ISR has the record.

Since Kafka 3.0 the producer defaults to acks=all (together with idempotence), so the defaults are already on the safe side.

min.insync.replicas: the floor

acks=all means "all of the ISR", and the ISR can shrink. If two of three replicas are dead, "all of the ISR" is only the leader, and acks=all would be no safer than acks=1. The topic (or broker) setting min.insync.replicas fixes this: if the ISR has fewer members than this number, the leader rejects acks=all writes with a NotEnoughReplicas error instead of accepting data that is not safely copied.

The classic safe combination is:

PROPERTIES
# topic / broker settings
replication.factor=3
min.insync.replicas=2

# producer
acks=all

With these, one broker can fail and writes continue (the ISR still has two members), and every acknowledged record exists on at least two brokers. If a second broker fails, the producer gets errors, which trades availability for not losing data. With min.insync.replicas=1 (the default), writes would keep succeeding on a lone leader.

A partition and its replicas, simulated

The stand-in below models one partition with three brokers. It supports acks="all" with a min.insync.replicas check, tracks the ISR and high watermark, and lets you kill brokers. Followers here copy the leader's log instantly when replication is requested, which is simpler than real fetch loops but shows the rules.

JS Replicas, ISR, high watermark and min.insync.replicas

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Follow the story. With three healthy brokers, each acks=all write is held by all three, and the high watermark is 2. After broker 3 dies, the ISR shrinks to {1, 2}, which still meets the minimum of 2, so writes continue. After broker 2 dies, the ISR is only , below the minimum, so the next write is rejected: the partition stays readable but refuses writes that could not be made safe. Broker 3's copy is behind (it never saw c), which is why it could not safely become leader later without catching up.

What is real here: the roles, the ISR shrinking, the high watermark as the minimum log end, and the NotEnoughReplicas behavior. What is simplified: real followers pull in the background rather than being copied on demand, and the leader waits with timeouts rather than instantly.

Failure and leader election

When a leader dies, the cluster's controller picks a new leader from the ISR. Because every ISR member holds all committed records, no acknowledged-with-acks=all data is lost. The new leader may hold a few records beyond the old high watermark that were never committed; followers use a leader epoch (a number that increases with every leadership change) to truncate any log suffix that diverged from the new leader.

The second stand-in compares what happens to acknowledged data when the leader crashes in two setups. Followers lag behind in the acks=1 case, because they only copied the first record.

JS Leader crash: acks=1 versus acks=all

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With acks=1, the producer was told that y and z were stored, but only the leader had them. When it crashed, they were gone. With acks=all, each record was acknowledged only after the followers had it, so the new leader holds everything. This is the central reason to use acks=all for data you cannot afford to lose.

Unclean leader election

What if every ISR member is gone, for example when the leader and the in-sync followers all crash, but an out-of-sync follower is still alive? Kafka then has two choices:

  • Wait for an ISR member to come back. The partition is unavailable meanwhile, but no committed data is lost. This is the default (unclean.leader.election.enable=false).
  • Elect the out-of-sync replica (an unclean election). The partition is available again quickly, but the new leader is missing records that were committed, and those are lost forever. Other replicas may even have to discard data to match it.

Choose per topic: a metrics topic may prefer availability, while an order or payment topic should not. Most deployments leave unclean election off.

Replication protects against broker failure, not against mistakes. If a producer writes bad data or a topic is deleted, every replica faithfully copies that change. Replication is not a backup.

Putting the settings together

Goal Typical settings
No acknowledged data loss, tolerate one broker down replication.factor=3, min.insync.replicas=2, acks=all
Maximum speed, loss is acceptable acks=1 or acks=0, no durability promise
Writes stay available if only the leader is left min.insync.replicas=1 (weaker guarantee)
Never lose committed data, even at the cost of availability unclean.leader.election.enable=false

Note the arithmetic: with a replication factor of 3 and a minimum of 2, you tolerate one failure for writes. A minimum of 3 (equal to the factor) would stop writes as soon as any one replica is down, which is rarely what you want.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Where do producers normally send their writes for a partition?

    All writes for a partition go through its leader, and followers copy from the leader.
  2. 2. A topic has replication.factor=3 and min.insync.replicas=2. The ISR drops to one replica. What happens to an acks=all write?

    With fewer ISR members than the minimum, the leader refuses acks=all writes instead of accepting data that is not safely replicated.
  3. 3. What is the high watermark?

    It is the smallest log end offset among ISR members. Records below it are committed; consumers cannot read beyond it.
  4. 4. Why can acks=1 lose acknowledged data?

    The producer is told success after only the leader wrote the record. If the leader fails before replication, the new leader never had it.
  5. 5. What is the risk of unclean leader election?

    Electing an out-of-sync replica restores availability but loses the records it never received.

Summary

  • Each partition has a leader that takes writes and followers that copy its log; a replication factor of 3 is typical.
  • The ISR is the set of replicas that are caught up; only ISR members can become leader by default.
  • The high watermark is the highest offset replicated to the whole ISR, and consumers read only up to it.
  • acks=all plus min.insync.replicas=2 with replication factor 3 survives one broker failure without losing acknowledged data.
  • Unclean leader election trades committed data for availability and is off by default.
  • Our simulations are simplified stand-ins; real followers fetch asynchronously with timeouts and leader epochs.

Next: Writing a Producer, where you send records from code.