Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท State and Fault Tolerance

Checkpoints and Barriers

Understand how Flink checkpoints work: barriers flow with the data, operators snapshot their state, sources remember offsets, and a failure rewinds the whole job consistently.

Flink jobs run for weeks. Machines fail, networks hiccup and deployments restart processes, so a job must be able to resume without losing events and without counting any twice. Flink's answer is the checkpoint: a consistent snapshot of the state of all operators plus the read position of every source, taken regularly while the job keeps running.

This lesson explains the mechanism with barriers, shows the configuration, and gives runnable Java stand-ins that reproduce the recovery guarantee. It follows Keyed State in Flink and State Types, TTL and Broadcast State.

Real versus stand-in. The configuration snippets are real Flink usage, read-only. The Java programs are teaching simulations of the barrier algorithm; they are not the Flink runtime.

The idea: a snapshot that lines up with the data

If each operator saved its state at an arbitrary moment, the pieces would not fit together: the counter might include an event that the source thinks it has not read yet. Flink avoids that with barriers, tiny markers injected into the stream.

flowchart LR
    S["Source<br/>offset 5"] --> A["Map"]
    A --> C["Count per key<br/>state"]
    C --> K["Sink"]
    CC["Checkpoint<br/>coordinator"] -. "trigger n" .-> S
    S -. "barrier n flows with records" .-> A
    A -.-> C
    C -.-> K
    C -- "snapshot state" --> ST[("Durable storage")]
    S -- "snapshot offset" --> ST

The coordinator asks sources to start checkpoint n. The barrier travels with the records; each operator snapshots when it sees it.

The steps are:

  1. The checkpoint coordinator (in the JobManager) triggers checkpoint n.
  2. Each source records its current position (for Kafka, the partition offsets) and emits a barrier n into the stream.
  3. Every operator, when the barrier arrives, snapshots its state to durable storage and forwards the barrier.
  4. When all operators including sinks have acknowledged, the checkpoint is complete.

Records before the barrier are included in the snapshot, records after it are not. That makes the snapshot equal to "the state after processing exactly the records up to the source offsets". This is a variant of the Chandy-Lamport algorithm for distributed snapshots.

Configuring checkpoints

Java
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(30_000);                               // every 30 s

CheckpointConfig cfg = env.getCheckpointConfig();
cfg.setCheckpointStorage("s3://my-bucket/flink/checkpoints");
cfg.setMinPauseBetweenCheckpoints(10_000);                     // breathing room for the job
cfg.setCheckpointTimeout(120_000);                             // give up on a slow one
cfg.setTolerableCheckpointFailureNumber(3);
cfg.setExternalizedCheckpointRetention(ExternalizedCheckpointRetention.RETAIN_ON_CANCELLATION);

The same settings exist as options in the cluster configuration (execution.checkpointing.interval, execution.checkpointing.timeout, execution.checkpointing.min-pause, and state.checkpoints.dir). The interval is a trade-off: short intervals mean little work to redo after a failure but more snapshot overhead; for many jobs, 30 seconds to a few minutes is a good start.

Recovery: rewind everything together

When a task fails, Flink stops the affected tasks (or the whole job, depending on the failover strategy), restores every operator's state from the latest completed checkpoint, and resets sources to the saved offsets. The records between the checkpoint and the failure are read again and reprocessed. Because the state was rolled back to the same instant, no record is counted twice inside Flink's state. This is called exactly-once state consistency. Whether the outside world sees each result once depends on the sink, which is the topic of Exactly-Once Sinks and Two-Phase Commit.

The simulation below runs a counting job with a checkpoint every four records and a crash in the middle. Compare the final result of a run with recovery against a run without any crash.

Java Stand-in: checkpoint, crash and recover with exact counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Despite the crash, the final counts match the clean run. Records 5 and 6 were processed twice, once before the crash and once after, but the state those first attempts produced was thrown away, so nothing was double counted.

Barrier alignment with several inputs

An operator with two input channels (after a keyBy shuffle, or a union) can receive barrier n on one channel while the other channel is still delivering records from before the barrier. To keep the snapshot consistent, the operator aligns: it stops reading the channel whose barrier has arrived (buffering its later records) until the barrier shows up on the other channel. Only then does it snapshot and forward the barrier.

Alignment is cheap when streams are balanced and can hurt when one channel is slow, because the fast channel waits. Under backpressure that waiting makes checkpoints take very long. Flink offers two relief valves: unaligned checkpoints, where the barrier overtakes in-flight records and those records are saved as part of the snapshot, and buffer debloating, which shrinks in-flight data. Both are options in CheckpointConfig, and unaligned checkpoints trade larger snapshots for faster, more predictable checkpoint times under pressure.

Java Stand-in: aligning a barrier across two input channels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Practical advice

  • Choose a checkpoint interval that is a small fraction of your acceptable recovery time, and set a minimum pause so checkpoints do not run back to back.
  • Watch checkpoint duration, size and alignment time in the web UI (see Monitoring and Backpressure); growing durations are usually a symptom of backpressure or skew.
  • Make sure user functions and state can be serialized; non-serializable fields in your function class fail at submission time.
  • Retain checkpoints on cancellation if you may want to resume from them, and prefer savepoints for planned upgrades (State Backends and Savepoints).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a checkpoint barrier do?

    Barriers travel with the records. When one reaches an operator, everything before it belongs to this snapshot and the operator saves its state.
  2. 2. After a failure, what does Flink restore?

    A consistent rewind needs every operator and every source to return to the same instant, so everything is restored together.
  3. 3. Why does barrier alignment slow checkpoints under backpressure?

    An operator with several inputs must see the barrier on all of them before snapshotting, so slow inputs delay the whole checkpoint.
  4. 4. What is the effect of unaligned checkpoints?

    Unaligned checkpoints skip waiting by persisting the in-flight data, at the cost of more I/O and larger checkpoints.

Summary

  • A checkpoint is a consistent snapshot of all operator state and all source offsets, built by barriers that flow with the data.
  • After a failure Flink restores the latest completed checkpoint and replays the records after it, giving exactly-once state.
  • Configure interval, storage, minimum pause, timeout and tolerable failures; watch duration and size.
  • Alignment across several inputs can delay checkpoints under backpressure; unaligned checkpoints and buffer debloating help.

Next: State Backends and Savepoints