Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 39 ยท Storage: Retention and Compaction

Retention, Segments and Disk Layout

See how Kafka stores partitions as segment files with indexes, how time and size retention delete old data, and what tiered storage changes, with runnable stand-ins.

A Kafka topic is not kept forever by default, and it is not one giant file either. Each partition is a directory of segment files, and old data disappears by deleting whole segments once they are older than the retention limit. Understanding that layout explains many surprises: why disk space is not freed the moment a record expires, why a new consumer can find an offset quickly, and why changing retention is cheap.

This lesson builds on The Commit Log Idea and Topics and Partitions.

Real versus stand-in. The directory listings and settings below describe real Kafka behavior and are read-only, because the site cannot run a broker. The TryIt blocks are plain JavaScript stand-ins that model segment rolling and retention rules with tiny numbers. Real defaults are far larger.

One partition on disk

.log The records themselves.
.index Offset to file position.
.timeindex Timestamp to offset.

Files are named after the first offset they hold.

1 / 2

Partition directories and segments

On a broker, every partition has a directory inside one of the folders listed in log.dirs. The name is the topic plus the partition number. A partition of orders that has seen some traffic might look like this:

TEXT
orders-0/
  00000000000000000000.log
  00000000000000000000.index
  00000000000000000000.timeindex
  00000000000000052310.log
  00000000000000052310.index
  00000000000000052310.timeindex
  00000000000000104871.log        <- active segment (being written)
  00000000000000104871.index
  00000000000000104871.timeindex
  leader-epoch-checkpoint

The log is split into segments. The number in each file name is the base offset, the offset of the first record in that segment, padded to 20 digits. Only the newest segment, the active segment, receives writes. When it grows past log.segment.bytes (1 GiB by default) or gets older than log.roll.ms/segment.ms (7 days by default), Kafka closes it and starts a new one whose base offset is the next offset to be assigned. Closed segments are never modified again.

Alongside each .log file are two sparse indexes:

  • The offset index maps an offset to a byte position in the .log file. It holds an entry only every few KB of data (log.index.interval.bytes, 4096 by default), not one per record.
  • The time index maps a timestamp to an offset, which is how tools and clients answer "where were we at 14:00?" (offsetsForTimes in the Java consumer).

To read offset N, the broker picks the segment with the largest base offset not greater than N (a binary search over file names), uses the index to jump close to N and scans forward a short distance. That keeps reads fast regardless of partition size, and writes stay sequential appends.

Rolling and lookup, simulated

This stand-in uses a segment size of three records so you can watch rolling happen. It names segment files the way Kafka does and finds a record with the same floor search.

JS Segment rolling and offset lookup

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Retention: how data expires

The default cleanup policy for a topic is delete. Kafka periodically (every log.retention.check.interval.ms, 5 minutes by default) checks closed segments against two limits:

Setting (topic / broker) Default Meaning
retention.ms / log.retention.hours 7 days Delete a segment when its newest record is older than this.
retention.bytes / log.retention.bytes -1 (no limit) Delete the oldest segments while the partition is larger than this.

Three details matter in practice:

  1. Whole segments only. A segment is eligible when its largest timestamp is older than the limit. A single old record inside a segment that also holds recent data stays. With the 1 GiB default, a quiet topic can keep data much longer than retention.ms because the active segment is not eligible at all until it rolls.
  2. The size limit is per partition, not per topic. A topic with 12 partitions and retention.bytes=1GB can use roughly 12 GB, before replication.
  3. If both limits are set, either one can delete. Space is not freed instantly: a deleted segment is renamed with a .deleted suffix and removed after file.delete.delay.ms (a minute by default).

Records are never changed by retention, only dropped from the front of the log. The first offset that still exists is the log start offset. A consumer whose committed offset is below it gets an out-of-range error and falls back to auto.offset.reset (see Reading with a Consumer). This is one way slow or long-stopped consumers lose data, so make retention longer than the longest outage you want a consumer to survive.

Retention in action

The stand-in applies both rules to four segments. Time is measured in days from "now = day 10".

JS Time and size retention

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last line shows that the active segment survives even with a very short time limit, matching real behavior. Lowering segment.ms or segment.bytes makes data expire closer to the configured retention, at the price of more, smaller files.

Changing retention

Retention is a per-topic override of the broker defaults, and you can change it on a live topic:

Terminal
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name orders \
  --add-config retention.ms=259200000,segment.bytes=268435456

This sets 3 days of retention and 256 MiB segments. The change applies at the next retention check, and shrinking retention can delete a lot of data at once. Setting retention.ms=-1 keeps data forever, which is legitimate for event-sourcing topics provided you budget disks for it.

Tiered storage

Keeping months of data on broker disks is expensive. Tiered storage (KIP-405, available in recent Kafka releases, check the documentation for your version) lets a topic keep only recent segments on local disks and move closed segments to a cheaper remote store such as object storage, through a pluggable RemoteStorageManager. A topic enables it with remote.storage.enable=true; local.retention.ms or local.retention.bytes says how much stays local, while retention.ms governs the total. Consumers keep the same API, and older reads simply get slower. Tiered storage is not available for compacted topics (see Log Compaction and Tombstones).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the number in a segment file name such as 00000000000000052310.log mean?

    Segments are named after their base offset, padded to 20 digits. That is what makes the floor search for an offset possible.
  2. 2. A topic has retention.ms of 1 hour, but a quiet partition's records are still there after a day. What is the most likely reason?

    The active segment is never deleted. With a 1 GiB or 7 day roll threshold, a quiet partition may not roll for a long time.
  3. 3. retention.bytes is set to 10 GB on a topic with 6 partitions. How much data can the topic keep (before replication)?

    retention.bytes applies to each partition separately, so six partitions can hold roughly six times the value.
  4. 4. Which statement about the offset index is true?

    The index adds an entry every few KB. The broker uses it to land close to the offset and scans the short remainder.

Summary

  • A partition is a directory of segment files named by base offset, each with an offset index and a time index; only the newest segment is written.
  • Segments roll by size (segment.bytes) or age (segment.ms); reads use a floor search by base offset plus a sparse index.
  • Retention deletes whole closed segments by age (retention.ms) or partition size (retention.bytes); the active segment is safe until it rolls.
  • Consumers that fall behind the log start offset lose data and fall back to auto.offset.reset.
  • Tiered storage moves old segments to remote storage to keep local disks small.

Next, a different way to keep data: Log Compaction and Tombstones.