Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 39 ยท Projects

Project: Build a Mini Kafka Broker

Build a miniature Kafka in one Node.js file: segmented partition logs, keyed produce, fetch by offset, consumer groups, generations and commits, with runnable code.

The best way to understand a system is to build a small version of it. In this project you write a toy Kafka broker in plain JavaScript: partition logs split into segments, a producer path that picks a partition by key, a fetch path that reads from an offset, and a group coordinator that assigns partitions and stores committed offsets. In the end a consumer crashes and a second one picks up exactly where the group left off.

It is about 130 lines, it runs entirely in your browser or on the site's Node runner, and each part mirrors a real Kafka idea from earlier lessons.

Real versus stand-in. This whole lesson is a stand-in. A real broker is written in Java and Scala, stores segments as files with sparse index files, speaks a binary network protocol, replicates between brokers and persists everything. Our mini broker keeps everything in memory in one process, has one replica, no network and no disk. It models the behavior you reason about as a Kafka user, not the implementation.

Four parts, one file

Partition log Segments of records with offsets.
Produce and fetch Key picks the partition.
Group coordinator Assigns partitions, bumps generations.
Committed offsets The group's saved progress.
1 / 2

Part 1: a partition is a list of segments

A partition is an ordered, append-only sequence of records, each with an offset (see The Commit Log Idea). A real broker stores it as a series of segment files named after their first offset (see Retention, Segments and Disk Layout): 00000000000000000000.log, then, once the first is full, a new file starting at the next offset. Reading from an offset means finding the right segment and scanning from there. Retention deletes whole old segments, never single records.

Our PartitionLog keeps a list of segments, each with a baseOffset and its records. append rolls a new segment when the active one is full. startOffset is the oldest offset still stored; endOffset is the offset the next record will get. This is the log end offset.

JS Part 1: the segmented partition log

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice what retention does: the first segment disappears, so startOffset jumps from 0 to 3, but the offsets of the remaining records do not change. Offsets are permanent names. A consumer asking for offset 0 now asks for something that no longer exists; a real consumer then follows its auto.offset.reset setting (see Reading with a Consumer). Our read simply returns what is still there.

The linear scan inside read is the part a real broker does better. It keeps a sparse offset index per segment, so it jumps close to the target and scans only a short stretch.

Part 2: topics, keys and the produce path

A topic is a list of partitions. produce hashes the key to choose one, so the same key always goes to the same partition (see Message Keys and Partitioners). Real Kafka uses murmur2; any stable hash teaches the same lesson. The broker returns the partition and the offset assigned, which is what a real producer callback receives in its record metadata.

fetch(topic, partition, offset, max) is the read path. Notice that the broker does not track who has read what. A fetch is a stateless request "give me records from this offset". That is why many consumer groups can read the same topic without affecting each other.

Part 3: the group coordinator

A consumer group splits a topic's partitions among its members, each partition to one member (see Consumer Groups and Rebalancing). Our coordinator keeps for each group a set of members, an assignment map and a generation number. Every membership change, a join or a leave, triggers a rebalance: the generation is increased and partitions are divided again. We use a range-style split, giving each sorted member a contiguous block of partitions.

The generation matters in two places. A consumer notices on its next poll that the group's generation has moved on, so it discards its local read positions and starts over from the committed offsets. And the broker rejects a commit that carries an old generation or comes from a member that left. That check is what stops a "zombie", a consumer that was kicked out but does not know it yet, from overwriting the progress of its replacement. Real Kafka has the same idea: commits carry the member's generation and a stale one fails.

Part 4: committed offsets

The committed offset for a partition is the next offset the group wants to read, so it equals the number of records fully handled when counting from zero (see Offsets and Commits). We store them in a Map keyed by group, topic and partition, a miniature __consumer_offsets. Lag is the log end offset minus the committed offset for each partition (see Monitoring, Lag and Troubleshooting).

A consumer's position (what it will fetch next) and the committed offset (what survives a restart) are two different things. The consumer advances the position as it polls and only moves the committed offset when commit() is called.

Put it together

The program below is the complete mini broker plus a scenario. It creates a three-partition topic with tiny two-record segments, produces twelve keyed events, and then:

  1. c1 joins alone, owns all three partitions, polls two records from each and commits.
  2. c2 joins. The group rebalances: c1 keeps partitions 0 and 1, c2 gets partition 2.
  3. c1 polls again but crashes (leaves) before committing.
  4. c2 now owns everything, resumes from the committed offsets and so reads the records c1 had polled but never committed.
  5. A late commit from the crashed c1 is rejected, and the lag drops to zero.
Node.js The complete mini broker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output from the top. Twelve records spread as four per partition, and partition 0 has two segments, based at offsets 0 and 2. Alone, c1 reads offsets 0 and 1 of every partition and commits [2, 2, 2]. After c2 joins (generation 2) the ownership splits. c1 then polls offsets 2 and 3 of partitions 0 and 1, but never commits them. When it leaves, c2 owns all three partitions and starts from the committed offset 2 everywhere. It therefore reads p0@2 p0@3 p1@2 p1@3 again, which c1 had already seen. That repeat is at-least-once delivery in miniature (see At-Most-Once, At-Least-Once, Exactly-Once). Partition 2 is different: c2 reads p2@2 p2@3, which nobody had seen before, so that is new work, not a repeat.

The final line shows the fence: the crashed c1 tries to commit offset 99 and the broker refuses.

Ideas to extend it

Each of these maps onto a lesson in this tutorial, so they make good experiments.

  • Replication. Give every partition a leader and a follower log, with an in-sync set and a high watermark (see Replication, Leaders and ISR). Consumers may only read up to the high watermark.
  • Retention. Call deleteOldSegments when a time or size limit passes, and make poll handle a committed offset below startOffset with a reset policy.
  • Compaction. Add a cleaner that keeps only the newest record per key in closed segments (see Log Compaction and Tombstones).
  • Heartbeats. Make members expire if they do not heartbeat within a session timeout, instead of leaving explicitly.
  • Idempotent producer. Give each producer an id and sequence number per partition and reject duplicates (see Batching, Compression and Idempotence).
  • Cooperative rebalancing. Change #rebalance so that members keep partitions that do not need to move.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the mini broker, why does retention delete whole segments instead of single records?

    Real Kafka does the same: it deletes whole old segment files, so the remaining records keep their offsets.
  2. 2. What does the committed offset of a partition mean for a group?

    It is the group's saved progress. A consumer that takes over a partition starts reading there.
  3. 3. Why does the broker reject a commit from the crashed consumer c1?

    A commit must carry the current generation and come from a current member. This fences zombie consumers.
  4. 4. In the scenario, why does c2 read some records that c1 had already polled?

    Position and committed offset are different. Only committed progress survives a consumer failure, so uncommitted records are delivered again.

Summary

  • A partition is a list of segments of offset-addressed records; retention removes whole old segments and offsets never change.
  • The key picks the partition, and produce returns the partition and offset.
  • Fetch is stateless: the broker serves records from any offset, so groups do not affect each other.
  • The group coordinator assigns partitions and bumps a generation on every membership change; stale generations cannot commit.
  • Committed offsets are the group's durable progress; uncommitted work is redelivered after a failure.
  • A real broker adds disk files with indexes, a network protocol, replication, compaction, heartbeats and much more, but the model is the same.

Next, test yourself with Practice: Exercises with Solutions.