Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 37 ยท Streaming Concepts

Delivery Guarantees and Exactly-Once

Learn at-most-once, at-least-once and exactly-once guarantees, how Flink checkpoints and barriers give exactly-once state, and why end-to-end needs idempotent or transactional sinks.

Machines fail. When a streaming job restarts, a hard question appears: which events were already processed? Process one twice and a bank balance is wrong; skip one and a sale disappears. Stream engines describe their answer as a delivery guarantee, and Flink offers the strongest one for its state: exactly-once.

This lesson defines the three guarantees, shows how checkpoint barriers achieve exactly-once state, and explains the difference between state and end-to-end guarantees, which is the most misunderstood point in streaming. It continues from Parallelism, keyBy and Key Groups.

The three guarantees

  • At-most-once. Every event is processed zero or one times. After a failure, in-flight events are simply lost. Fast and simple; acceptable only for things like best-effort metrics.
  • At-least-once. Every event is processed one or more times. After a failure the input is replayed, so some events are counted twice. No loss, but duplicates.
  • Exactly-once. The effect of every event is applied once, as if there had been no failure.

Be careful with the phrase. Events may physically be processed more than once after a restart (they are replayed from the last checkpoint); exactly-once means the state and outputs end up as if each was processed once.

Checkpoints and barriers

Flink periodically takes a checkpoint, a consistent snapshot of all operator state plus the position in each source. It does so without stopping the stream, using the Chandy-Lamport idea with checkpoint barriers:

sequenceDiagram
    participant JM as JobManager
    participant S as Source
    participant O as Operator
    participant K as Sink
    JM->>S: trigger checkpoint 7
    S->>S: snapshot offset
    S->>O: barrier 7 inside the stream
    O->>O: snapshot state when barrier 7 arrives
    O->>K: barrier 7
    K->>JM: ack
    O->>JM: ack
    JM->>JM: checkpoint 7 complete

A barrier flows with the records. Each operator snapshots its state when the barrier reaches it.

  1. The JobManager tells the sources to start checkpoint n.
  2. Each source records its position (for Kafka: the offsets) and injects a barrier n into the stream. The barrier separates records before the snapshot from records after it.
  3. When an operator receives barrier n on all its input channels, it snapshots its state (to the state backend and then to durable storage such as S3 or HDFS) and forwards the barrier. If one input is faster, its records after the barrier are buffered until the slower inputs catch up. This alignment is what makes the snapshot consistent.
  4. When every operator has acknowledged, checkpoint n is complete.

After a failure, Flink restores all state from the last complete checkpoint and resets the sources to the stored positions. Events after the checkpoint are replayed and update the restored state once.

Practical settings:

Java
env.enableCheckpointing(60_000);                                       // every 60 s, EXACTLY_ONCE by default
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(30_000);
env.getCheckpointConfig().setCheckpointTimeout(10 * 60_000);
env.getCheckpointConfig().setCheckpointStorage("s3://my-bucket/checkpoints");

Choosing AT_LEAST_ONCE skips barrier alignment, which lowers latency but can leave duplicates in state after recovery. Newer versions also offer unaligned checkpoints that let barriers overtake buffered records to cope with backpressure. These are covered in the chapter on state and fault tolerance.

Stand-in: replay with and without snapshots

Java Stand-in: at-least-once versus exactly-once state after a crash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The at-least-once variant overcounts the events that were applied before the crash and then replayed. Restoring state and position from the same checkpoint gives the true counts.

State exactly-once versus end-to-end exactly-once

Flink's checkpoints guarantee that its own state reflects each event once. But a job also writes to the outside world, and Flink cannot roll back a database row that it already wrote. After a restart, events from the last checkpoint are processed again and the sink may receive the same result twice.

So end-to-end exactly-once needs cooperation from the source and the sink:

  • Replayable source. The source must be able to re-read from a position (Kafka offsets, file offsets). A source that cannot replay (a plain socket) allows only at-most-once or at-least-once.
  • Idempotent sink. Writing the same record twice leaves the same result, for example an upsert by primary key (INSERT ... ON CONFLICT UPDATE) or overwriting a file name derived from the data.
  • Transactional sink with two-phase commit. Records are written inside a transaction that is pre-committed when a checkpoint barrier arrives and committed only when the checkpoint is known to be complete. If the job fails, uncommitted transactions are aborted. The Kafka sink (with DeliveryGuarantee.EXACTLY_ONCE), and the file sink work like this.

Readers of a transactional sink must read only committed data (isolation.level=read_committed for Kafka consumers), and the results appear with a delay of up to one checkpoint interval.

Stand-in: an idempotent sink

Java Stand-in: duplicates are harmless when the sink upserts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The append sink shows duplicates after the replay; the upsert sink ends up correct. This is why a primary key on your output table is one of the cheapest ways to get end-to-end correctness.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. After a failure, input is replayed and some events are processed again. What does exactly-once mean here?

    Replays do happen, but state is restored from the checkpoint, so each event's effect is applied once.
  2. 2. What does a checkpoint barrier do?

    Barriers travel with the data; an operator snapshots when the barrier arrives on all its inputs.
  3. 3. Which is needed for end-to-end exactly-once in addition to checkpoints?

    Flink cannot undo writes to external systems, so the sink must tolerate replays or commit transactionally.
  4. 4. Which sink approach makes duplicate writes harmless?

    An upsert by key stores the same final row however many times it is written.

Summary

  • At-most-once may lose events, at-least-once may duplicate them, exactly-once applies each event's effect once.
  • Flink takes consistent snapshots with checkpoint barriers and restores state and source positions together.
  • State exactly-once is built in; end-to-end exactly-once also needs a replayable source and an idempotent or two-phase-commit sink.
  • Transactional sinks make output visible only after the checkpoint completes.

Next: Backpressure and Flow Control.