Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 36 ยท Streaming

Delivery Guarantees and Deduplication

Understand at-most-once, at-least-once and exactly-once delivery, why duplicates happen, and how idempotent sinks and deduplication make streams effectively exactly-once.

In a stream, machines crash and networks drop messages while events keep flowing. A system must decide what to promise when something fails mid-flight: may an event be lost, may it be processed twice, or neither? These promises are the delivery guarantees, and understanding them tells you why a "clean" stream often needs a deduplication step.

Real brokers (Kafka, Kinesis) and engines (Flink, Spark Structured Streaming) implement these guarantees with transactions and checkpoints. Those cannot run here, so the examples use a small simulated pipeline with an injected crash. The behaviour you see is the behaviour of the real systems at the consumer-offset level.

The three guarantees

Guarantee Meaning Failure behaviour
At-most-once Each event processed zero or one time A crash can lose events
At-least-once Each event processed one or more times A crash can cause duplicates
Exactly-once Each event has its effect exactly once No loss and no visible duplicates

The difference comes from when you commit the offset relative to when you do the work. This is the whole mechanism:

  • Commit before processing: if you crash after committing but before processing, the event is skipped, which is at-most-once.
  • Commit after processing: if you crash after processing but before committing, the event is processed again on restart, which is at-least-once.
Python Simulate a crash between work and commit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With commit-first, e3 is lost. With commit-after, e3 appears twice. Most pipelines choose at-least-once, because losing data is usually worse than seeing it twice, and then remove the duplicates.

Why exactly-once is subtle

True "exactly once" delivery over an unreliable network is impossible in general, since a sender cannot always know whether a message arrived. What systems provide is exactly-once processing semantics (effects happen once) by combining at-least-once delivery with one of:

  1. Idempotent writes: applying the same event twice has the same effect as once.
  2. Transactions: the output write and the offset commit succeed or fail together.
  3. Deduplication: remember which events were already handled and skip repeats.
flowchart LR
    A["At-least-once delivery"] --> B["Idempotent sink<br/>or dedup"]
    B --> C["Effectively exactly-once"]

Duplicates become harmless.

Technique: idempotent sink

If the sink is keyed by event id, repeated writes overwrite themselves. Here is the earlier at-least-once run writing into a keyed store:

Python At-least-once plus an idempotent sink

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The keyed sink reports 100, the append-only list 130. A relational equivalent is INSERT ... ON CONFLICT DO NOTHING or an upsert, as in Idempotent Pipeline Design.

Technique: stateful deduplication

When the sink cannot be made idempotent (for example counting into an aggregate), deduplicate before it. Keep a set of recently seen event ids. The set cannot grow forever, so bound it by a time window matching how long duplicates can plausibly occur.

Python Dedup with an expiring id window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first a is accepted and the repeat at t=3 is dropped; the a at t=20 is accepted again because the id expired from memory. That is the trade-off: a bigger retention window catches later duplicates but costs more state. Always give events a stable unique id at the source (a UUID or a deterministic hash of the business key); without one, dedup is guesswork.

Technique: transactional output

Some platforms let a processor write its output and commit its input offset atomically, so a crash rolls back both. Simulate the idea with a database transaction that updates a result row and an offset row together:

Python Output and offset in one transaction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The crash happened after the total was updated but before the offset moved. Because both were in one transaction, the update was rolled back and the retry applied it once. The total is exactly 24.

Choosing in practice

  1. Default to at-least-once delivery.
  2. Make sinks idempotent (upsert by event id) wherever possible.
  3. Where you must count or aggregate, deduplicate by a stable id within a bounded window, or use transactional writes if the platform supports them.
  4. Monitor duplicates and loss: count events in and out, and alert when they diverge.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Committing the offset before processing gives which guarantee?

    If the worker crashes after the commit but before processing, the event is never processed.
  2. 2. Committing the offset after processing gives which guarantee?

    A crash between processing and commit means the event is processed again.
  3. 3. What is the most common way to achieve effectively exactly-once results?

    Duplicates are tolerated in delivery but made harmless in effect.
  4. 4. Why does an id-based dedup set need an expiry window?

    State must be bounded; the window should match how late duplicates can plausibly arrive.
  5. 5. What does a transactional write of output plus offset achieve?

    Atomic commit of result and progress prevents partial effects.

Summary

  • At-most-once can lose events, at-least-once can duplicate them, and exactly-once is achieved by combining at-least-once with idempotence, deduplication or transactions.
  • The order of offset commit versus processing decides the guarantee.
  • Give every event a stable unique id; make sinks idempotent; bound dedup state with a retention window.
  • Monitor inputs against outputs to catch loss or duplicates.

Next: Data Engineering track overview, for the remaining chapters on storage, modeling, orchestration and governance.