Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท State and Fault Tolerance

Keyed State in Flink

Learn why stream processing needs state, how Flink scopes state to the current key with ValueState, and how to model it with runnable Java stand-ins.

A stream processor that only looks at one event at a time is easy to build and rarely useful. The interesting questions all need memory: how many clicks has this user made today, what was the previous temperature of this sensor, has this order id been seen before? In Flink the memory of an operator is called state, and Flink manages it for you: it stores it efficiently, saves it in checkpoints, restores it after a crash and redistributes it when you change the parallelism.

This lesson covers the most common kind, keyed state, and the ValueState handle you use to read and write it. Chapter 3 showed how to build a pipeline with the DataStream API; here we give that pipeline a memory.

Real versus stand-in. The site cannot run a Flink cluster, so the Flink code blocks in this lesson are real API usage shown read-only. The runnable examples are small Java programs that model the same ideas with plain collections. They are not Flink.

Why state?

Counts Events so far per user.
Previous value Compare with the last reading.
Dedup Remember ids already seen.

Without state every event is a stranger.

1 / 2

Keyed streams and the current key

State in Flink is keyed when the stream was split with keyBy. All events with the same key are sent to the same parallel task, and while your function processes an event, Flink sets a hidden "current key". Every state handle you call then silently refers to the value of that key.

Java
DataStream<Reading> readings = ...;

readings
    .keyBy(Reading::sensorId)          // partition by sensor
    .process(new SpikeDetector())      // a KeyedProcessFunction
    .print();

You never pass the key to the state object. If sensor A and sensor B both call lastValue.value(), each gets its own answer. That single idea replaces a whole class of manual Map<String, ...> bookkeeping and is also what makes the state movable: Flink knows which key owns which bytes.

ValueState in a KeyedProcessFunction

ValueState<T> holds one value per key. You get a handle in open(), using a descriptor that gives the state a name and a type, and you use it in processElement.

Java
public class SpikeDetector extends KeyedProcessFunction<String, Reading, String> {

    private transient ValueState<Double> lastValue;

    @Override
    public void open(OpenContext ctx) {
        lastValue = getRuntimeContext().getState(
            new ValueStateDescriptor<>("last-value", Types.DOUBLE));
    }

    @Override
    public void processElement(Reading r, Context ctx, Collector<String> out) throws Exception {
        Double previous = lastValue.value();          // null the first time for this key
        if (previous != null && Math.abs(r.value() - previous) > 10.0) {
            out.collect("spike on " + r.sensorId() + ": " + previous + " -> " + r.value());
        }
        lastValue.update(r.value());                  // remember for the next event
    }
}

Points to notice:

  • The handle field is transient and is created in open(), never in the constructor, because the function object is serialized and shipped to the cluster before it runs.
  • value() returns null when nothing was stored for the current key. Always handle that first case.
  • update(x) replaces the value and clear() removes it. Clearing state you no longer need is how you keep memory bounded.
  • The name "last-value" is part of the stored data. Renaming it later makes old checkpoints and savepoints unreadable for that state (see State Backends and Savepoints).

Our Java stand-in below recreates the contract: one table of values, and a "current key" that every call uses. Run it and watch each sensor keep its own history.

Java Stand-in: ValueState scoped by the current key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Sensor B jumps from 5 to 30 and A from 21.5 to 40, and neither event is confused with the other sensor's readings, even though they are interleaved in the input.

A running count per key

Counting is the classic first stateful job. You can write it by hand with ValueState<Long>, or let Flink's built-in aggregations (sum, reduce) hide the state for you. Writing it by hand shows what those helpers do.

Java
@Override
public void processElement(Click click, Context ctx, Collector<Tuple2<String, Long>> out) throws Exception {
    Long current = count.value();
    long next = (current == null ? 0L : current) + 1;
    count.update(next);
    out.collect(Tuple2.of(click.userId(), next));
}

Because state is part of the job's fault tolerance, a crash does not reset the counts to zero. After a restart Flink restores the state from the last completed checkpoint and rewinds the source to the matching position. The program above is the same code before and after a failure.

Java Stand-in: a hand-written counter next to a built-in style helper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The first loop prints a new running total for each event, like a Flink job does: a stream of updates, not one final answer. The second loop shows the final table.

Why state must stay small and bounded

Keyed state grows with the number of distinct keys. A job that stores one Long per user is fine; one that stores every event for every user forever is a slow memory leak that will eventually kill the job or make checkpoints enormous. Three habits prevent this:

  1. Store aggregates (a count, a sum, the latest value) rather than raw events whenever you can.
  2. Call clear() when a key's work is finished, for example when a session ends.
  3. Use state time-to-live and timers for keys that may never come back. The next lesson, State Types, TTL and Broadcast State, shows how.

Operator state (not keyed) exists too, mainly inside sources and sinks that remember offsets or buffered records, but ordinary application code uses keyed state almost always.

Reading or writing keyed state outside a keyed context fails at runtime. State handles only work after keyBy, inside functions such as KeyedProcessFunction, RichFlatMapFunction and KeyedCoProcessFunction.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What decides which value ValueState.value() returns inside processElement?

    Flink sets a hidden current key for each event after keyBy, and every state handle reads and writes the entry of that key.
  2. 2. Where should a state handle be initialized?

    The function is serialized and shipped to the cluster, so handles must be created at runtime in open(). Creating them per call is not possible either, because the runtime owns them.
  3. 3. What does value() return the first time a key is seen?

    Nothing has been stored for that key yet, so value() returns null. Handle the null case explicitly.
  4. 4. Which habit keeps keyed state bounded?

    State grows with the number of keys and the data kept per key. Aggregating, clearing and expiring keep it from growing forever.

Summary

  • State is the memory of an operator; Flink stores, checkpoints, restores and rescales it.
  • Keyed state is scoped by the current key that keyBy establishes, so each key sees its own value.
  • ValueState<T> has value(), update() and clear(); create the handle in open() from a named descriptor.
  • A hand-written counter and a built-in sum do the same thing; state survives failures because it is checkpointed.
  • Keep state small: aggregate, clear and expire.

Next: State Types, TTL and Broadcast State