Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Connectors and Deployment

Exactly-Once Sinks and Two-Phase Commit

See how Flink achieves end-to-end exactly-once: replayable sources, checkpointed state and transactional or idempotent sinks with a two-phase commit protocol, with runnable simulations.

"Exactly-once" is one of the most misunderstood phrases in stream processing. Flink's checkpoints guarantee that state reflects each record exactly once (Checkpoints and Barriers). But records that were already written to an external system before a crash are not in any checkpoint, so replaying the stream after recovery could write them a second time. End-to-end exactly-once closes that gap with sinks that cooperate with checkpoints.

Real Flink code is read-only here; the Java programs simulate the failure scenarios so you can see the guarantee appear and disappear.

Three ingredients

End-to-end exactly-once needs all three:

  1. A replayable source (Kafka offsets, file positions) so records can be read again.
  2. Checkpointed state, so the job rewinds to a consistent point.
  3. A sink that is either idempotent or transactional, so replayed output does not show up twice.

If one piece is missing the guarantee degrades to at-least-once: nothing is lost, but duplicates are possible.

flowchart LR
    S["Replayable source"] --> P["Stateful operators<br/>checkpointed"]
    P --> K["Idempotent or<br/>transactional sink"]
    K --> X[("External system")]

The chain is only as strong as its weakest link.

Idempotent sinks

A sink is idempotent if writing the same record twice has the same effect as writing it once. Upserting a row by primary key is idempotent (INSERT ... ON CONFLICT DO UPDATE), and so is writing a file named after the record's id. After a replay, the duplicates overwrite themselves. The weak spot: for a short time after recovery, readers may see an older value that is then replaced, so it is "effectively once" with a brief inconsistency.

The first simulation compares an append-only sink with an upsert sink when a replay happens after a crash.

Java Stand-in: replay after a crash with append versus upsert sinks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The append sink holds k3 and k4 twice. The upsert sink is unchanged.

Transactional sinks and two-phase commit

When the target cannot upsert (a plain Kafka topic, a file system), Flink uses a two-phase commit (2PC) protocol integrated with checkpoints. The sink writes into an open transaction and only makes it visible once the checkpoint that covers the data has completed everywhere.

The phases, using the TwoPhaseCommitSinkFunction idea (the modern Sink API expresses the same steps as Committer and SinkWriter):

  1. Begin: when a new checkpoint interval starts, open a transaction (for Kafka, a producer transaction); write records into it.
  2. Pre-commit (on the checkpoint barrier): flush the data and prepare the transaction, but do not commit. The transaction id is saved into the checkpoint state.
  3. Commit (when the JobManager notifies that checkpoint n is complete for all operators): commit the transaction, making the data visible.
  4. Abort: if the checkpoint fails, or after a restore to an earlier checkpoint, abort transactions that were never committed.

On recovery from checkpoint n, Flink restores the saved transaction ids and re-commits transactions belonging to completed checkpoints (commit must therefore be idempotent), and aborts the rest. Uncommitted data from the failed attempt is never seen by readers using read-committed isolation.

Java
KafkaSink<Order> sink = KafkaSink.<Order>builder()
    .setBootstrapServers("broker:9092")
    .setRecordSerializer(...)
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("orders-v1")          // unique per job and stable across restarts
    .setProperty("transaction.timeout.ms", "900000") // must exceed checkpoint interval plus downtime
    .build();

Consumers of that topic must set isolation.level=read_committed, otherwise they also see uncommitted and aborted records and the guarantee is lost for them. Also note the latency cost: output becomes visible only at checkpoint completion, so the checkpoint interval is the lower bound on end-to-end latency for exactly-once sinks.

The next program simulates the whole cycle with a "database" that has a pending transaction area. Run it, then compare the crash-before-commit and commit-then-crash cases.

Java Stand-in: two-phase commit across checkpoints

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

In both scenarios each record appears exactly once in the committed output, regardless of where the crash falls.

What can still go wrong

  • Transaction timeout. If the job is down longer than the broker's transaction.timeout.ms, prepared transactions expire and data is lost; set it generously and keep recovery fast.
  • External side effects. Calling an HTTP API or sending an email from a map function is not transactional. Make such calls idempotent (send an idempotency key) or move them to a sink with its own guarantees.
  • Non-replayable sources. A socket or a message queue that deletes on read cannot be replayed, so the chain is at-most-once or at-least-once at best.
  • Savepoint compatibility. Keep transactionalIdPrefix and the sink's uid stable across upgrades so the saved transactions can be resumed (State Backends and Savepoints).

Choosing a strategy

Target Strategy
Relational database with a key Idempotent upsert (JDBC upsert)
Kafka topic Transactional KafkaSink with EXACTLY_ONCE and read_committed consumers
Files / object store FileSink (commit at checkpoint)
Arbitrary REST API Idempotency keys and at-least-once
Dashboards where duplicates do not matter At-least-once, cheapest and lowest latency

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are checkpoints alone not enough for end-to-end exactly-once?

    State is rolled back, but data already sent to an external system is not, unless the sink is idempotent or transactional.
  2. 2. When does a two-phase commit sink make its data visible?

    Pre-commit happens on the barrier; the commit is triggered when the checkpoint is complete for all operators.
  3. 3. What must downstream Kafka consumers use to benefit from an EXACTLY_ONCE sink?

    read_committed consumers ignore uncommitted and aborted transactional records.
  4. 4. What is a cost of exactly-once transactional sinks?

    Results only become visible when a checkpoint completes.

Summary

  • End-to-end exactly-once needs a replayable source, checkpointed state and an idempotent or transactional sink.
  • Idempotent sinks (upserts by key) absorb replays; transactional sinks use two-phase commit tied to checkpoints.
  • Phases: begin, pre-commit on the barrier, commit on checkpoint completion, abort on failure; commits must be idempotent on recovery.
  • Mind transaction timeouts, external side effects, stable transactional ids and read_committed consumers.
  • Latency is bounded below by the checkpoint interval; use at-least-once when duplicates are acceptable.

Next: Deployment Modes and Clusters