Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท State and Fault Tolerance

State Types, TTL and Broadcast State

Choose between ValueState, ListState, MapState and reducing state, expire old entries with state TTL, and share rules to all tasks with broadcast state in Flink.

ValueState is the simplest handle, but a real job often needs more: a list of recent events per user, a small map of counters per device, or a rule set that every parallel task must see. Flink offers a family of state primitives for these cases, a way to let old entries expire, and a special pattern called broadcast state for data that is not keyed at all.

This lesson builds on Keyed State in Flink. Flink code is shown read-only; the runnable blocks are Java stand-ins that model the behaviour.

The keyed state primitives

All of these are scoped to the current key, created in open() through a descriptor and checkpointed automatically.

Handle Holds, per key Typical use
ValueState<T> one value last reading, a flag, a counter
ListState<T> a list you append to recent events, a buffer for a window
MapState<K, V> a small map counters per sub-category, a per-user feature table
ReducingState<T> one value, folded with a ReduceFunction running sum or maximum
AggregatingState<IN, OUT> an accumulator with a different output type running average

ListState and MapState matter for performance when you use the RocksDB backend (next lesson but one). Those handles read and write individual elements instead of deserializing the whole collection, so a MapState with thousands of entries per key stays cheap, whereas a ValueState<HashMap<...>> would copy the entire map on every access.

Java
private transient MapState<String, Long> perPage;

@Override
public void open(OpenContext ctx) {
    perPage = getRuntimeContext().getMapState(
        new MapStateDescriptor<>("views-per-page", Types.STRING, Types.LONG));
}

@Override
public void processElement(Click c, Context ctx, Collector<String> out) throws Exception {
    Long n = perPage.get(c.page());
    perPage.put(c.page(), n == null ? 1L : n + 1);
}

The stand-in below implements a ListState that keeps the last three events per key, a very common "sliding buffer" pattern, and a MapState of counters.

Java Stand-in: ListState as a bounded buffer and MapState as counters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Keeping state from growing: state TTL

Imagine a job that remembers the last order status for every customer. Customers who never return still occupy space forever. State time-to-live (TTL) tells Flink to forget an entry when it has not been touched for a given time.

Java
StateTtlConfig ttl = StateTtlConfig
    .newBuilder(Duration.ofDays(7))
    .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
    .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
    .build();

ValueStateDescriptor<String> desc = new ValueStateDescriptor<>("last-status", Types.STRING);
desc.enableTimeToLive(ttl);

The important choices are:

  • Update type. OnCreateAndWrite restarts the clock only when the value is written; OnReadAndWrite also restarts it on reads.
  • Visibility. NeverReturnExpired hides an expired value even if it has not been physically removed yet; ReturnExpiredIfNotCleanedUp may still return it.
  • Cleanup strategy. Expired entries are removed lazily: when read, when a full snapshot is taken, and, with RocksDB, during compaction. TTL uses processing time, so a job that was stopped for days does not instantly lose all its state because event time jumped.

TTL is a safety net for state size, not a business rule. If an expiry must trigger an action (for example "close the session after 30 minutes of silence"), use a timer in a KeyedProcessFunction instead, which can emit output.

Java Stand-in: expiring entries with a manual clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice that order-1 was only physically removed when we asked for it, the lazy cleanup that real Flink also uses.

Broadcast state

Keyed state is partitioned: each task holds only its keys. Sometimes every task needs the same small data set, for example a list of fraud rules or a feature-flag table that changes while the job runs. Broadcast state solves this. You connect a normal stream with a second stream that is broadcast to every parallel instance. Each instance stores the broadcast elements in an identical map and reads it while processing the main stream.

Java
MapStateDescriptor<String, Rule> rulesDesc =
    new MapStateDescriptor<>("rules", Types.STRING, TypeInformation.of(Rule.class));

BroadcastStream<Rule> ruleStream = rules.broadcast(rulesDesc);

transactions
    .keyBy(Transaction::accountId)
    .connect(ruleStream)
    .process(new KeyedBroadcastProcessFunction<String, Transaction, Rule, Alert>() {
        @Override
        public void processElement(Transaction t, ReadOnlyContext ctx, Collector<Alert> out) throws Exception {
            for (Map.Entry<String, Rule> e : ctx.getBroadcastState(rulesDesc).immutableEntries()) {
                if (e.getValue().matches(t)) out.collect(new Alert(t, e.getKey()));
            }
        }
        @Override
        public void processBroadcastElement(Rule r, Context ctx, Collector<Alert> out) throws Exception {
            ctx.getBroadcastState(rulesDesc).put(r.name(), r);   // runs on EVERY instance
        }
    });

Two rules keep broadcast state consistent. The main-stream side gets read-only access, and only processBroadcastElement may modify the map, because each instance must apply the same updates in the same way. And because the order of events across two streams is not defined, results can differ slightly between runs around the moment a rule changes.

Java Stand-in: two tasks holding the same broadcast rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which handle is best for a per-key table of thousands of counters when using RocksDB?

    MapState lets RocksDB read and write single entries. A ValueState with a HashMap would serialize the whole map on each access.
  2. 2. What does StateVisibility.NeverReturnExpired do?

    Cleanup is lazy, so an expired value may still sit in storage. This setting guarantees your code never sees it.
  3. 3. You must emit an alert when a user has been silent for 30 minutes. What should you use?

    TTL silently removes data and cannot produce output. A timer fires a callback in which you can emit the alert and clear state.
  4. 4. Why may the main-stream side of a broadcast connection only read the broadcast state?

    Each parallel instance keeps its own copy. If main-stream elements (which differ per instance) could change it, the copies would diverge.

Summary

  • Keyed state comes in ValueState, ListState, MapState, ReducingState and AggregatingState; prefer MapState and ListState over large values.
  • State TTL expires entries that are untouched for a period, using lazy cleanup and processing time.
  • Use timers when expiry must trigger output; use TTL as a size safety net.
  • Broadcast state shares a small, changing data set (rules, flags) with every parallel task; only the broadcast side can write it.

Next: Checkpoints and Barriers