Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 39 ยท Consumers

Offsets and Commits

Learn how Kafka consumers track progress: committed offsets, __consumer_offsets, auto vs manual and sync vs async commits, seek and replay, and consumer lag, with runnable stand-ins.

A partition is a numbered sequence of records, and the number of a record is its offset. A consumer reading the partition has a position: the offset of the next record it will receive. As long as the consumer is running, it remembers its position in memory. But what happens after a crash or a deployment? The new process needs to know where the old one stopped. That memory is the committed offset, saved in Kafka itself.

How and when you commit decides what happens during failures: whether records are processed twice, or skipped, or neither. This lesson explains the mechanics. The next one, At-Most-Once, At-Least-Once, Exactly-Once, turns them into guarantees.

Real versus stand-in. The consumer API calls and tool commands are real and shown read-only (a cluster is not available on the site). The TryIt blocks are plain JavaScript stand-ins that keep a log, a position and a committed offset the way a consumer group does, so you can crash and restart them.

Position versus committed

Position Next record this consumer will get.
Committed offset Where a restarted consumer resumes.
Log end offset Next offset a producer will write.

Lag is the log end offset minus the committed offset.

1 / 2

What a committed offset means

A commit stores, for one group and one partition, the offset of the next record to read. After processing records 0 to 6, you commit 7. A restarted consumer in the same group asks for the committed offset and continues at 7. This off-by-one is the source of many bugs: the committed value is "last processed plus one".

The offsets live in a compacted internal topic called __consumer_offsets. A commit is simply a record whose key is (group, topic, partition) and whose value is the offset, so the newest commit for a key wins (see Log Compaction and Tombstones). The group coordinator broker (see Consumer Groups and Rebalancing) serves them. Offsets of groups that have been empty for a long time expire (seven days by default, offsets.retention.minutes), after which auto.offset.reset applies again.

Because commits belong to the group, different groups track their own positions in the same topic independently.

Auto commit

With enable.auto.commit=true (the default), the consumer commits automatically every auto.commit.interval.ms (5 seconds by default). It happens inside the next call to poll, and it commits the offsets returned by the previous poll. That is convenient, but it has a hidden cost: your code does not control the timing. If the process dies after a commit but before the records were truly handled, those records are skipped. If it dies after handling but before the next auto-commit, they are replayed. Auto commit is fine for simple, tolerant applications and risky when every record matters.

Manual commits

Switch auto commit off and call commit yourself, after the work is done:

Java
props.put("enable.auto.commit", "false");

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
        process(record);
    }
    consumer.commitSync();   // commit positions returned by this poll
}

commitSync() blocks until the broker answers and retries on temporary errors. It is safe but adds a round trip per batch. commitAsync() returns immediately and calls your callback later; it does not retry, because a late retry could overwrite a newer commit with an older value. A common pattern combines them:

Java
try {
    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) process(r);
        consumer.commitAsync();             // fast path in the loop
    }
} finally {
    try {
        consumer.commitSync();              // one reliable commit on the way out
    } finally {
        consumer.close();
    }
}

You can also commit specific offsets, for example after every 100 records inside a large batch, by passing a map from partition to new OffsetAndMetadata(offset + 1). Remember the plus one.

Crash, restart, redelivery

The stand-in below processes a 12-record partition in batches of 4 and commits after each batch. The process crashes in the middle of the second batch, after handling two of its records but before committing. A new instance of the same group starts from the committed offset.

JS Commit, crash, and redelivery

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Records r4 and r5 were handled by the first instance but not committed, so the second instance handled them again. That is at-least-once behaviour: nothing is lost, but your handler may run twice for the same record. Committing before processing would flip the failure mode and skip records instead. Either way, you choose the trade-off with the position of the commit call. The consequences, and how to cope with duplicates, are the subject of the next lesson.

Notice also what batch size does. A bigger batch means fewer commits but a bigger replay window after a crash.

Seek and replay

The position is not locked to the committed offset. A running consumer may move it with:

  • seek(partition, offset) to jump to a specific offset,
  • seekToBeginning(partitions) and seekToEnd(partitions),
  • offsetsForTimes(map) to find the first offset at or after a timestamp, then seek to it.

Because Kafka keeps records after they are read (until retention removes them), this is how you replay: re-run a bug fix over yesterday's data, rebuild a search index, or reprocess a bad hour. For a stopped group you can also reset offsets with the command line tool:

Terminal
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group invoice-service --topic orders \
  --reset-offsets --to-datetime 2026-01-15T00:00:00.000 --dry-run

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group invoice-service --topic orders \
  --reset-offsets --to-earliest --execute

--dry-run shows what would change; --execute applies it. The group must have no active members when you reset.

Consumer lag

Lag for a partition is the log end offset minus the group's committed offset: how many records exist that the group has not yet confirmed. It is the most watched consumer metric. A steady small lag is normal; a growing lag means the consumers are slower than the producers.

Terminal
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group invoice-service

The output lists, for each partition, the current offset (committed), the log end offset and the lag. Monitoring is covered in Monitoring, Lag and Troubleshooting.

The stand-in below holds a small three-partition topic with timestamps. It computes lag per partition, then seeks back to a timestamp to replay and shows the lag jump.

JS Lag, seek and replay by time

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Seeking back raises the lag because more records are now waiting to be read; seeking to the end drops it to zero because the consumer skips everything that exists. Both are legitimate tools, and both change what your application will process.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A consumer has processed records up to offset 41. What should it commit?

    A committed offset is the next offset to read, so last processed (41) plus one.
  2. 2. Why does commitAsync not retry failed commits?

    Async commits can complete out of order. Retrying an older one after a newer one succeeded would move the group's progress backwards.
  3. 3. A consumer using auto commit dies right after the client committed, but before the records of that poll were processed. What happens?

    The offsets were already committed, so the replacement consumer starts after them and those records are never handled.
  4. 4. How is consumer lag for a partition computed?

    Lag counts the records that exist in the partition beyond what the group has committed.
  5. 5. Where are committed offsets stored?

    Commits are records in __consumer_offsets, keyed by group, topic and partition, so the latest commit per key is the group's position.

Summary

  • A committed offset is the next offset to read, stored per group and partition in __consumer_offsets.
  • Auto commit is convenient but timing is not under your control; manual commits after processing give at-least-once behaviour.
  • commitSync is reliable and blocks; commitAsync is fast and does not retry; combine them, finishing with a sync commit on shutdown.
  • seek, seekToBeginning, offsetsForTimes and kafka-consumer-groups.sh --reset-offsets let you replay or skip.
  • Lag is log end offset minus committed offset and is the first thing to watch.

Next, see what these choices mean for losing or repeating data in At-Most-Once, At-Least-Once, Exactly-Once.