Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 37 ยท Streaming Concepts

Stateless and Stateful Processing

Learn the difference between stateless and stateful operators in Flink, what keyed state and operator state are, why state matters, and how it stays safe in failures.

Some operations only need the current event. Others need to remember what happened before. That distinction, stateless versus stateful, decides how hard an operator is to scale and to recover, and it is the reason Flink has a whole machinery for managed state.

You have seen both already: in Setting Up Flink and Running Your First Job flatMap was stateless and sum(1) was stateful. Here we look at why that matters.

Stateless operators

A stateless operator computes its output from the current record alone. Run it on any subtask, in any order, restart it at any time: the result for a record never changes.

  • map: convert Celsius to Fahrenheit.
  • filter: keep events with amount > 0.
  • flatMap: split a sentence into words.

Stateless operators scale trivially. If a subtask dies, Flink restarts it and nothing is lost except in-flight records, which are replayed from the source.

Stateful operators

A stateful operator's output depends on earlier records, so it has to store something between events.

  • A running count or sum per user.
  • The last seen value to detect a change.
  • A window that has collected events but not yet fired.
  • A session with the pages a visitor viewed so far.
  • Deduplication: the set of IDs already seen.
  • Pattern detection: "login failed three times in five minutes".
  • Machine-learning models or reference tables held in memory.

This is where streaming gets hard. The state is shared with time: it keeps growing as long as the job runs, it must survive a crash, and it must be split across machines when you scale up. Flink solves this by managing the state for you instead of leaving it to variables in your own objects.

Why not use a plain Java map?

Imagine you keep counts in a HashMap field inside your function. Three problems appear immediately:

  1. Loss on failure. If the JVM dies, the map is gone and counts restart at zero.
  2. No rescaling. If you change parallelism from 2 to 4, you must redistribute the map entries yourself, and keys must still land on the same subtask as their events.
  3. No consistency. After a restart, input is replayed from the source, so a map that survived (for example in an external DB) would count some events twice.

Flink-managed state fixes all three: it is checkpointed together with the read positions of the sources, so the state and the input offsets always agree, and it is organised so that it can be redistributed.

Two kinds of managed state

Keyed state is available after keyBy. Each key has its own private state, and your function only sees the state of the key of the current record. Types include ValueState, ListState, MapState, ReducingState and AggregatingState. Keyed state is split into key groups for rescaling, explained in Parallelism, keyBy and Key Groups.

Operator state belongs to a parallel subtask of an operator and is not tied to a key. Sources use it to remember their read offsets, and sinks to remember buffered, not-yet-committed data. Its two redistribution styles are even split (list state is divided when parallelism changes) and union (every subtask gets the full list).

A third kind, broadcast state, copies the same data (for example a rule set) to every parallel instance.

Later chapters cover the APIs in full; see the State and Fault Tolerance chapter of this track.

Java
public class RunningTotal extends KeyedProcessFunction<String, Order, String> {
    private transient ValueState<Double> total;

    @Override
    public void open(OpenContext ctx) {
        total = getRuntimeContext().getState(new ValueStateDescriptor<>("total", Types.DOUBLE));
    }

    @Override
    public void processElement(Order o, Context ctx, Collector<String> out) throws Exception {
        double sum = (total.value() == null ? 0.0 : total.value()) + o.amount();
        total.update(sum);
        out.collect(o.customer() + " has spent " + sum);
    }
}

total looks like a field, but each key (each customer) gets its own value, stored in the state backend, snapshotted at every checkpoint.

Stand-in: stateless versus stateful

Java Stand-in: a stateless filter and a stateful running total

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Stand-in: why recovery needs a snapshot

This simulation processes events, takes a "checkpoint" (a copy of the state plus the input position), crashes, and restores. Restoring state together with the position gives the right answer; restoring only the position (or only the state) does not.

Java Stand-in: state and input position must be restored together

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Only the first line is correct. This is the heart of Flink's exactly-once state guarantee: a checkpoint pairs the state with the input position, so replaying after a failure never double-counts or skips an event.

Design guidance

  • Prefer stateless operators whenever the logic allows it; they are the cheapest and the easiest to scale.
  • Keep state small and bounded: expire old keys with timers or state TTL, or it grows until the job runs out of disk or memory.
  • Store only what you need; a count is cheaper than a list of all events.
  • Choose the state backend for the size: heap-based for small state, RocksDB for state larger than memory.
  • Remember that state is per key; a global state (one key) becomes a single-subtask bottleneck.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which operator is stateless?

    A filter decides from the current record alone. The others must remember earlier records.
  2. 2. Why is a plain HashMap field in a function a bad place for counts?

    Only Flink-managed state is checkpointed and rescalable.
  3. 3. What is keyed state?

    Each key has its own private state values; the function sees the state of the current key.
  4. 4. Why does a checkpoint contain both state and source positions?

    State without its matching position (or the reverse) would count some events twice or not at all.

Summary

  • Stateless operators look at one record; stateful operators remember earlier records.
  • Plain Java fields are not safe state: use Flink-managed keyed state or operator state.
  • Keyed state exists per key after keyBy; operator state belongs to a parallel subtask; broadcast state is copied to all.
  • Checkpoints store state and source positions together, which makes recovery consistent.
  • Keep state small, expire it, and pick a backend that fits its size.

Next: Parallelism, keyBy and Key Groups.