Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท State and Fault Tolerance

State Backends and Savepoints

Pick between the HashMap and RocksDB state backends, enable incremental checkpoints, and use savepoints with stable operator uids to upgrade a Flink job safely.

Two questions remain after you know what state and checkpoints are. Where does state physically live while the job runs, and how do you stop a job, change it and continue with the same state? The first is decided by the state backend, the second by savepoints. Getting both right is the difference between a job you can operate for years and one you are afraid to touch.

Flink configuration and CLI commands below are real and read-only. The runnable parts are Java stand-ins for serialization and state restore. This lesson continues Checkpoints and Barriers.

Two state backends

A state backend controls how keyed state is stored at runtime and how it is written into a checkpoint.

HashMap backend RocksDB backend
Runtime storage Java objects on the heap Serialized bytes in an embedded RocksDB, off-heap and on local disk
Speed Very fast reads and writes (no serialization per access) Slower per access (serialize and deserialize), but predictable
State size limit Memory of the TaskManager Local disk size
Garbage collection Large heaps can cause long GC pauses Little GC pressure
Checkpoints Always full snapshots Full or incremental
Use when State is modest (up to a few GB per task) State is large (tens or hundreds of GB), or very many keys

You choose a backend by configuration, without changing your code:

YAML
# Flink configuration file
state.backend.type: rocksdb
state.backend.incremental: true
execution.checkpointing.interval: 60 s
state.checkpoints.dir: s3://my-bucket/flink/checkpoints
state.savepoints.dir: s3://my-bucket/flink/savepoints

or for one job in code:

Java
Configuration config = new Configuration();
config.set(StateBackendOptions.STATE_BACKEND, "rocksdb");
config.set(CheckpointingOptions.INCREMENTAL_CHECKPOINTS, true);
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);

Checkpoint storage (where snapshot files are written, for example S3, HDFS or a shared disk) is a separate choice from the backend. The backend decides how state lives at runtime; storage decides where durable copies go.

Incremental checkpoints

A RocksDB state is stored in immutable files (SSTables). An incremental checkpoint uploads only the files that are new since the previous checkpoint and refers to old ones, so a 200 GB state with 1 GB of daily change does not copy 200 GB every minute. The price is a slightly more complex recovery, since the checkpoint is a chain of files, and Flink cleans old files up only when no checkpoint refers to them.

The stand-in shows the idea with key-level diffs instead of SSTables: only changed keys go into each incremental snapshot, and a restore replays the chain.

Java Stand-in: full versus incremental snapshots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Savepoints: checkpoints you control

A checkpoint is created automatically for failure recovery, owned by Flink and usually deleted as newer ones complete. A savepoint is a snapshot you trigger yourself, which Flink never deletes. Use savepoints for planned operations:

  • upgrading your job code or the Flink version;
  • changing parallelism;
  • moving the job to another cluster;
  • A/B testing a logic change from the same starting state.
Terminal
# take a savepoint while the job keeps running
flink savepoint <jobId> s3://my-bucket/flink/savepoints

# stop with a savepoint: the cleanest way to stop a stateful job
flink stop --savepointPath s3://my-bucket/flink/savepoints <jobId>

# start the new version from it
flink run -s s3://my-bucket/flink/savepoints/savepoint-ab12cd-1234567890 \
    --detached new-version.jar

flink stop drains the job (sources stop, in-flight data finishes, the final savepoint is taken), so with a transactional sink there is no half-written output.

Giving operators stable identities

When Flink restores a savepoint, it must match each saved piece of state to an operator in the new program. It does this with operator uids. If you do not set one, Flink generates it from the position of the operator in the job graph, so adding or removing an operator in the middle of the pipeline can make Flink match the wrong state or fail to match any. Always set uids on stateful operators and sources and sinks:

Java
events
    .keyBy(Event::userId)
    .process(new SessionTracker()).uid("session-tracker")     // stable identity
    .name("Session tracker");                                 // label for the UI

If you remove an operator on purpose, restore with --allowNonRestoredState to tell Flink that dropping the old state is fine. Without that flag, a savepoint that contains state nobody wants fails the restore, which protects you from silent data loss.

The stand-in below implements the matching rule: state is saved by uid in a byte array, and a new job restores each operator whose uid appears in the snapshot.

Java Stand-in: savepoint bytes restored by operator uid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Real savepoints use Flink's own serializers rather than Java serialization, but the logic is identical: bytes keyed by uid, matched on restore.

State schema evolution

You will eventually want to add a field to a class stored in state. Flink supports schema evolution for state types that use Avro or POJO serialization, which handle added or removed fields and changed field order. Types that fall back to Kryo (generic serialization) do not evolve safely, so prefer POJOs, Avro, or Java records registered as POJOs for anything stored in state. This is worth checking before the job has a year of state in production; the web UI warns when a type uses Kryo.

Checkpoint or savepoint?

Need Use
Automatic recovery from crashes Checkpoint
Deploy a new version, rescale, migrate Savepoint
Smallest snapshot cost during normal running Incremental checkpoint
A portable, officially supported format across Flink versions Savepoint (canonical format)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which backend suits tens of gigabytes of state per task?

    RocksDB keeps state serialized on local disk, so the limit is disk size, and it avoids large-heap GC pauses. HashMap keeps objects on the heap.
  2. 2. What happens if you do not set operator uids and then insert an operator in the middle of the job?

    Without explicit uids Flink derives ids from the job graph structure, so changing the graph can change the ids.
  3. 3. Why use flink stop --savepointPath rather than cancel?

    Stop-with-savepoint finishes in-flight work and snapshots the final state, which is the safe way to hand a stateful job to a new version.
  4. 4. What does an incremental checkpoint upload?

    Incremental checkpoints exploit RocksDB's immutable files to avoid re-uploading unchanged data.

Summary

  • The state backend decides where state lives at runtime: HashMap on the heap (fast, memory-bound) or RocksDB on disk (large, steady, supports incremental checkpoints).
  • Checkpoint storage (where snapshots are written) is configured separately.
  • Savepoints are manually triggered, durable snapshots for upgrades, rescaling and migration; use flink stop --savepointPath.
  • Set .uid(...) on stateful operators, sources and sinks; use --allowNonRestoredState only when you intend to drop state.
  • Avoid Kryo-serialized types in state if you expect schema changes.

Next: Table API First Steps