Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 39 ยท Delivery Guarantees

At-Most-Once, At-Least-Once, Exactly-Once

Where Kafka duplicates and data loss come from, how commit ordering gives at-most-once or at-least-once, and how idempotent consumers and dedup keys approach exactly-once.

Every messaging system must answer an awkward question: when something fails halfway, is a message lost, delivered twice, or handled exactly once? Kafka does not hide the question. It gives you the tools, and your configuration plus your code decide the answer.

There are three classic guarantees. At-most-once means a record is handled zero or one times: it can be lost, never duplicated. At-least-once means one or more times: never lost, but possibly repeated. Exactly-once means the effect happens one time, even when the system retried underneath. This lesson locates the failure points in a pipeline, shows how commit ordering produces the first two, and shows how an idempotent consumer gets the effect of the third. Transactions and Exactly-Once covers Kafka's own transactional machinery.

Real versus stand-in. The settings and behaviours described are real Kafka, with config shown read-only. The TryIt blocks are plain JavaScript stand-ins with injected failures, so every run is repeatable. They model the ideas, not the real clients.

Three guarantees

At-most-once May lose records. Never repeats.
At-least-once Never loses. May repeat.
Exactly-once The effect happens one time.

Real systems build the third from the second plus care.

1 / 2

Where loss and duplicates come from

Follow one record through the pipeline.

  1. Producer to broker. With acks=0 or acks=1, a record can vanish if the leader dies before followers copy it (Replication, Leaders and ISR). If an acknowledgement is lost and the producer retries, the broker may hold the record twice, unless the producer is idempotent (Batching, Compression and Idempotence).
  2. Inside the cluster. acks=all with a sensible min.insync.replicas and no unclean leader election means an acknowledged record survives a broker failure.
  3. Broker to consumer. Reading is safe: records stay in the log. Nothing is "consumed away".
  4. Consumer processing and commit. Here the interesting trade-off lives. Handling a record and committing its offset are two separate actions, and a crash can fall between them.

The producer settings that give the strongest durability are:

PROPERTIES
acks=all
enable.idempotence=true
# topic or broker level:
min.insync.replicas=2
unclean.leader.election.enable=false

With those, the producer-to-broker path is lossless and duplicate-free for a single producer session. What remains is the consumer's side.

Commit ordering decides the failure mode

Take a consumer that polls a batch, handles each record and commits. Two orders are possible.

Commit first, then process (at-most-once). If the consumer crashes after committing but before finishing the work, the restarted consumer begins after those records. They are lost to the application.

Process first, then commit (at-least-once). If the consumer crashes after the work but before the commit, the restarted consumer begins at the old committed offset and handles those records again. They are duplicated, never lost.

Auto commit behaves somewhere between the two, depending on timing, which is why serious pipelines commit manually (Offsets and Commits).

The stand-in below runs an identical 10-record workload and an identical crash (the process dies while handling record 6, once) for both orders. The "database" counts how many times each record's effect was applied.

JS Commit before or after processing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Commit-first lost records 6 and 7 (the crash happened inside a batch that had already been committed), while process-first processed 4 and 5 twice. A bare consumer cannot avoid both mistakes at once: you must pick which one you can tolerate, or add more machinery.

Making duplicates harmless: idempotent consumers

If duplicates are unavoidable at-least-once, the usual answer is to make the handler idempotent: applying the same record twice has the same effect as applying it once. Some operations are naturally so:

  • SET balance = 120 instead of balance = balance + 20,
  • an upsert keyed by the record's business id,
  • writing to a file or object named by the record's id.

Others are not (sending an email, charging a card, incrementing a counter), and these need a deduplication key. Every record gets a unique identity: a business id in the value ("order 1001, event PAID"), or the triple topic, partition, offset. The consumer stores the ids it has already processed, in the same transaction as the effect, and skips any id it has seen:

SQL
BEGIN;
INSERT INTO processed_events (event_id) VALUES ('order-1001-paid');  -- unique key
-- if this violates the unique constraint, roll back and skip the record
UPDATE accounts SET balance = balance - 20 WHERE id = 7;
COMMIT;

The important property is that the effect and the "I've seen this" marker are saved atomically. If they were separate, a crash between them would bring the problem back.

An even tidier trick is to store the consumed offset in the same database as the result, and on startup seek to that stored offset instead of using Kafka's committed offset. Then the offset and the data can never disagree.

JS At-least-once plus a dedup key

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Without the key, record e3 is applied twice and account A ends with 110 instead of 80. With the key, the replay of e3 is recognised and ignored, and the totals are exactly right even though Kafka delivered the record twice. This is effectively exactly-once: the delivery is at-least-once, the processing is exactly-once in effect.

What about "exactly-once" in Kafka itself?

Kafka offers exactly-once semantics for pipelines that read from Kafka and write back to Kafka. A consumer-transform-producer loop can commit its output records and its input offsets in one atomic transaction, and downstream consumers using isolation.level=read_committed only see committed results. That works because input position, output records and commit all live inside Kafka. It is the subject of Transactions and Exactly-Once.

The guarantee ends at the cluster's edge. As soon as your handler writes to a relational database, calls an HTTP API or sends an email, Kafka cannot make that effect transactional. For those, an idempotent consumer with a dedup key (or an idempotency key sent to the external API) is the practical tool.

Choosing a guarantee

Situation Typical choice
Metrics, logs, sampled telemetry where loss is tolerable at-most-once, or auto commit
Most business events at-least-once with idempotent handlers
Kafka-to-Kafka stream processing needing exact counts Kafka transactions
Writes to a database at-least-once plus dedup in the same DB transaction

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A consumer commits the offset first and then processes the batch, and crashes mid-batch. Which guarantee does this give?

    The offset was already moved forward, so unprocessed records are skipped on restart. Records may be lost but are never repeated.
  2. 2. Which handler is naturally idempotent?

    Writing the same row to the same key twice leaves the same state. The others change the world each time they run.
  3. 3. Why must the dedup marker and the business effect be saved in the same transaction?

    If the effect is saved but the marker is not, the replay repeats the effect. If the marker is saved but the effect is not, the effect is lost. Atomicity avoids both.
  4. 4. Which pair of settings gives the most durable producer-to-broker path?

    Waiting for the in-sync replicas with a minimum of two and de-duplicating retries covers both loss and duplicates on the way in.

Summary

  • Loss and duplication arise at the producer (acks, retries), in replication, and mostly at the consumer's process-versus-commit step.
  • Commit before processing gives at-most-once; process before commit gives at-least-once.
  • Idempotent handlers and deduplication keys, saved atomically with the effect, turn at-least-once delivery into exactly-once effects.
  • Kafka's own exactly-once covers Kafka-to-Kafka pipelines; external systems need idempotence on your side.
  • Pick the weakest guarantee your business can tolerate and design for it explicitly.

Next, see Kafka's transactional machinery in Transactions and Exactly-Once.