Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 39 ยท Getting Started

The Commit Log Idea

Understand the append-only commit log behind Kafka: offsets, immutability, replay, retention and why sequential disk writes make Kafka fast.

Under all of Kafka's features sits one very simple data structure: the commit log. A log is an ordered list of records where you can only add to the end. Every record gets the next number in line. If you understand this structure, most of Kafka's behavior stops being surprising: why readers can replay, why records cannot be edited, why ordering works only inside one partition, and why Kafka is fast on ordinary hard disks.

This lesson builds a log from scratch in JavaScript and then connects it to what the real broker does.

Append-only, numbered, immutable

A log has three rules.

Three rules of the log

Append only New records go to the end. Nothing is inserted in the middle.
Numbered Each record gets the next offset: 0, 1, 2, and so on.
Immutable A stored record is never changed.

Old records leave only when retention deletes the oldest ones.

1 / 2

The number given to a record is its offset. The first record in a partition has offset 0, the next 1, and so on. Offsets never change and are never reused for a different record: if record 5 is deleted by retention, no new record will ever be number 5. A reader's position is just "the next offset I want", so a consumer can remember where it stopped with a single integer.

Because records are immutable, there is no "update" in Kafka. If a customer changes their address, you append a new event. If you need to correct a mistake, you append a correction. This makes the log a faithful history, and it means readers never see a record change underneath them.

A log in 25 lines

Here is a minimal log. It uses a private field to hold records, freezes each record, and supports appending and reading from an offset. It also tracks an earliest offset, because a real log eventually deletes its oldest records.

JS A tiny append-only log

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the results. Appending returns the offset. Reading from offset 1 returns up to three records. Reading from offset 5 returns an empty list, because 5 is the next offset to be written: a consumer that has caught up simply gets nothing and waits. Trying to modify a record has no effect, because the record is frozen (in strict mode the assignment throws instead of being ignored, and the code above handles both cases).

Kafka calls the "next offset to be written" the log end offset. Real Kafka also has a log start offset (what earliest models here) and, with replication, a high watermark that marks which records are safely copied. You will meet that in replication and ISR.

Replay: the superpower

Since the log keeps its history, a consumer can read it again from any offset. This is a big difference from a queue. It enables:

  • Recovery. A service crashed and lost its state? Rebuild it by replaying events.
  • New consumers. A new analytics job starts at offset 0 and processes history.
  • Bug fixes. If a consumer had a bug, fix it, reset its position and re-run.
  • Many independent views. Each consumer derives its own view from the same events.

The next stand-in keeps an account history as events and rebuilds the balance by replaying from different offsets. It also shows what happens after retention deletes old records.

JS Replay and retention

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Replaying from offset 0 gives the full balance, 100. A consumer that starts at offset 2 only sees the later events, so it computes a different number (30): a reader that wants correct state must start where its state begins, or use a snapshot. The lag numbers show a very useful measure you will use often: lag is how many records a reader is behind the end of the log.

When retention removes old records, replay from before the new earliest offset becomes impossible. Real Kafka returns an "offset out of range" error and the consumer's auto.offset.reset setting decides what happens next (reading with a consumer).

Why a log is fast

Kafka stores each partition as files and only ever adds to the end of the newest file. That access pattern is friendly to hardware.

  • Sequential I/O. Writing and reading in order is much faster than jumping around a disk. Even spinning disks do well at sequential throughput, and SSDs are better still. A database updating records in random places cannot match that pattern.
  • The operating system page cache. Kafka leans on the OS to cache recently written files in memory. A consumer reading fresh data usually reads from RAM, not disk.
  • Batching. Producers send many records in one request and brokers write them as one chunk, so per-record overhead is small.
  • Zero-copy transfer. On the read path the broker can send file data to the network socket without copying it through application memory, when no TLS encryption is in the way.
  • Simple bookkeeping. The broker does not track which consumer read which message. Readers keep an offset, which is one number.

None of these ideas is exotic. The speed comes from choosing a data structure that matches how disks and operating systems like to work.

"Fast" here means high throughput: very many records per second moving through the system. It does not mean a single record arrives in microseconds. Kafka trades a little latency, through batching and replication, for throughput and durability.

What is real and what is a stand-in

In this lesson, the Log class keeps records in a JavaScript array. A real Kafka partition is a directory of segment files on disk, with index files that map offsets to file positions, and records are copied to follower brokers. The behavior you saw is real: append returns the next offset, readers keep a position, records are immutable, and retention removes only the oldest data. Segment files are covered in retention and segments.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A partition currently has records at offsets 0 to 9. What offset will the next appended record get?

    Offsets increase by one for each appended record. The ten records use 0 to 9, so the next one is 10. This is also the log end offset.
  2. 2. How do you "change" a stored record in Kafka?

    Records are never edited. Corrections and updates are modeled as new records appended to the end of the log.
  3. 3. What is consumer lag?

    Lag is the distance between the log end offset and the offset a reader will read next. It shows how far behind a consumer is.
  4. 4. Why is appending to a log efficient for hardware?

    Sequential reads and writes are much faster than random ones, and the operating system caches recent file data in memory.

Summary

  • A commit log is append-only, numbered by offset and immutable.
  • A reader's position is a single offset, so many readers can read the same log independently.
  • Replay means reading again from an earlier offset; retention eventually deletes the oldest records.
  • Lag is the distance between a reader's position and the log end.
  • Kafka is fast because sequential I/O, the page cache, batching and simple bookkeeping fit hardware well.
  • Our Log class is a stand-in; real partitions are segment files that are replicated to other brokers.

Next: Your First Topic with the CLI, where you run a broker and use Kafka's command-line tools.