Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 39 ยท Storage: Retention and Compaction

Log Compaction and Tombstones

Learn how Kafka log compaction keeps the latest value per key, how tombstones delete keys, and how compacted topics act as tables, with a runnable stand-in.

Time-based retention answers "keep the last seven days". Some topics need a different promise: "keep the latest value for every key, however old". Think of a topic holding each customer's current email address, or a product's current price. Old versions are noise; only the newest matters, yet you do not want the key to vanish just because it has not changed for a month.

That is log compaction. A compacted topic is still an ordinary append-only log, but a background cleaner removes records that a newer record with the same key has replaced. This lesson explains the rules, how to delete a key with a tombstone, and why compacted topics are the foundation for treating a log as a table. It builds on Retention, Segments and Disk Layout.

Real versus stand-in. The topic settings below are real and read-only (the site cannot run a broker). The TryIt blocks are plain JavaScript stand-ins that copy the compaction rules on a small array. The real cleaner works segment by segment on disk, with threads, throttling and a map of keys in memory.

Compaction in one picture

Before A=1, B=1, A=2, A=3, B=2
Cleaner Drops records with a newer same-key record.
After A=3, B=2 (offsets keep their gaps)
1 / 2

Turning compaction on

Compaction is a cleanup policy, set per topic:

Terminal
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic customer-emails \
  --partitions 3 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \
  --config delete.retention.ms=86400000

cleanup.policy=compact replaces the default delete. You may also set compact,delete to compact and expire old data by time. Every record in a compacted topic must have a key, because the key is the identity that decides what replaces what; a record with a null key is rejected.

What the cleaner does

Each partition log is split in two for the cleaner:

  • The clean part has been compacted before: it holds at most one record per key.
  • The dirty part is everything written since, and can hold many records per key.

The log cleaner threads on a broker pick the partition whose dirty part is the largest share of the log (min.cleanable.dirty.ratio, default 0.5, is the threshold to start). They read the dirty part to build a map of key to latest offset, then rewrite segments, copying only records that are still the latest for their key. Several guarantees follow:

  • The active segment is never compacted, so the newest writes always stay as they are, and readers can see several versions of one key near the head of the log.
  • A consumer that reads from the beginning sees at least the final value for every key, in the original order, but not necessarily every intermediate value. Compaction is not a history of changes.
  • Offsets never change. Surviving records keep their original offsets, so offsets in a compacted partition have gaps. A consumer whose position is in a gap just continues with the next record.
  • min.compaction.lag.ms guarantees that a record stays uncompacted for at least that long, and max.compaction.lag.ms forces compaction to happen within a time bound.

Compaction, simulated

JS Keep the latest value per key

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Tombstones: deleting a key

How do you say "this customer no longer exists"? You write a record with the key and a null value, called a tombstone. When the cleaner sees a tombstone it removes all earlier records for that key, but it keeps the tombstone itself for a while so that consumers, including ones that are catching up or rebuilding state, also see the deletion.

That retention period is delete.retention.ms (24 hours by default). It starts counting once the tombstone is in the cleaned part of the log. After it elapses, the next compaction removes the tombstone and the key is gone entirely. The implication for consumers: if one is offline longer than delete.retention.ms and then reads only from its stored offset, it can miss a tombstone, so state rebuilds should read the whole topic or use a generous setting.

In code, a tombstone is just a record whose value is null (shown read-only):

Java
producer.send(new ProducerRecord<String, String>("customer-emails", "bob", null));

Tombstones in the stand-in

JS A tombstone removes a key

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A compacted topic as a table

If you read a compacted topic from the start and keep the latest value per key in a map, you rebuild the current state of a table: the key is the primary key, and a tombstone is a row delete. This is a stream-table duality, and it powers a lot of Kafka design:

  • A service restarts and restores its local cache by replaying the topic instead of querying a database.
  • Kafka Streams stores its state-store changelogs in compacted topics (see Kafka Streams and Stream Processing).
  • Kafka itself uses compaction for __consumer_offsets: the latest committed offset per group, topic and partition (see Offsets and Commits).
  • Change-data-capture pipelines publish one record per row key so that the topic mirrors the table (see Event-Driven Patterns).
JS Rebuild a table by replaying

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second line replays only the records that would survive a compaction (the first sku-1 price and the first sku-2 price are superseded, and sku-2's tombstone remains). The table comes out identical, but the log is shorter, and that equivalence is the whole point.

Things to watch

  • Compaction removes the history; if you need an audit trail, use a delete-policy topic or compact,delete with a long retention.
  • Compaction runs in the background and uses disk and CPU. Its timing is not exact, so never design logic that depends on a specific old record disappearing by a particular time.
  • A key with a high update rate and a huge dirty ratio makes the cleaner work harder; size log.cleaner.threads and memory accordingly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a compacted topic guarantee for each key?

    Compaction keeps at least the last record per key. Older versions may be removed, so it is not a full change history.
  2. 2. How do you delete a key from a compacted topic?

    A null value is a tombstone. The cleaner removes earlier records for the key and later removes the tombstone itself after delete.retention.ms.
  3. 3. After compaction, what happens to the offsets of the surviving records?

    Offsets are permanent identifiers. Compaction removes records but never renumbers the rest.
  4. 4. Why does Kafka keep a tombstone for delete.retention.ms instead of removing it at once?

    A reader that is behind must be able to learn that the key was deleted, so the tombstone stays for a grace period.

Summary

  • cleanup.policy=compact keeps at least the latest record per key; keys are required.
  • The cleaner works in the background on the dirty part of the log; the active segment is never compacted and offsets are never changed.
  • A record with a null value is a tombstone; it removes the key and is itself dropped after delete.retention.ms.
  • Replaying a compacted topic rebuilds a table: this is how caches, Kafka Streams state and __consumer_offsets work.
  • Compaction is not a history log; combine with delete when you need time-bounded history.

Next, make the data in these topics well-defined with Schemas, Avro, Protobuf and JSON.