Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 37 ยท DataStream API

Process Functions: State and Timers

Use Flink's low-level ProcessFunction: access records, keyed state and timers in one place to build inactivity alerts, deduplication and custom logic that built-in operators cannot do.

map, filter and rolling aggregations cover many jobs, but sooner or later you need something they cannot express: "alert me if a device sends nothing for five minutes", "emit only the first event per ID within an hour", "buffer events and release them when something else arrives". For these, Flink offers its most powerful building block: the process function.

A process function gives you three things at once: the record, keyed state and timers. Windows, joins and pattern matching are built on exactly this. It continues from keyBy, Reduce and Rolling Aggregations.

The family

  • ProcessFunction<I, O>: for a plain stream. Gives records, timestamps and side outputs, but no keyed state or timers (these need a keyed stream).
  • KeyedProcessFunction<K, I, O>: for a KeyedStream. Adds keyed state and timers. This is the one you will use most.
  • CoProcessFunction, KeyedCoProcessFunction, BroadcastProcessFunction: for two connected streams (Connect, Union and Side Outputs).
  • ProcessWindowFunction: runs per window with access to all window elements.

Anatomy of a KeyedProcessFunction

Java
public class InactivityAlert extends KeyedProcessFunction<String, Heartbeat, String> {
    private static final long LIMIT_MS = 5 * 60_000;
    private transient ValueState<Long> pendingTimer;

    @Override
    public void open(OpenContext openContext) {
        pendingTimer = getRuntimeContext().getState(new ValueStateDescriptor<>("pendingTimer", Types.LONG));
    }

    @Override
    public void processElement(Heartbeat beat, Context ctx, Collector<String> out) throws Exception {
        Long old = pendingTimer.value();
        if (old != null) {
            ctx.timerService().deleteProcessingTimeTimer(old);        // device is alive: reset
        }
        long fireAt = ctx.timerService().currentProcessingTime() + LIMIT_MS;
        ctx.timerService().registerProcessingTimeTimer(fireAt);
        pendingTimer.update(fireAt);
    }

    @Override
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
        out.collect("device " + ctx.getCurrentKey() + " silent for 5 minutes");
        pendingTimer.clear();
    }
}

heartbeats.keyBy(Heartbeat::deviceId).process(new InactivityAlert());

Important points:

  • processElement is called for each record. Context ctx exposes the record's timestamp (ctx.timestamp()), the current key (ctx.getCurrentKey()), the timerService() and side outputs (ctx.output(tag, value)).
  • onTimer is called when a registered timer fires. It runs with the same key that registered the timer, so keyed state in it refers to that key.
  • State lives in ValueState, ListState, MapState and others, created in open from a StateDescriptor. Access always refers to the current key.
  • Collector<O> out emits any number of outputs.

Timers

Timers are Flink's way of saying "wake me up later" without a thread per key. They are stored in state, checkpointed and rescaled with the keys.

  • Processing-time timers: fire according to the machine's wall clock (registerProcessingTimeTimer(t)).
  • Event-time timers: fire when the watermark passes t (registerEventTimeTimer(t)), so they follow the data's own clock; see Watermarks.
  • Timers are deduplicated per key and timestamp: registering the same time twice fires once.
  • deleteProcessingTimeTimer / deleteEventTimeTimer cancel a timer.
  • Many timers mean much state; coalescing timestamps (for example to the full second or minute) reduces cost.
sequenceDiagram
    participant E as Event
    participant P as processElement
    participant T as Timer service
    E->>P: heartbeat for device 7
    P->>T: register timer now+5min, remember it in state
    E->>P: another heartbeat
    P->>T: delete old timer, register new one
    Note over T: no heartbeat for 5 min
    T->>P: onTimer fires
    P->>P: emit alert, clear state

The inactivity alert: every heartbeat pushes the timer into the future; silence lets it fire.

Stand-in: a keyed process function with timers

The stand-in implements a small timer service and an inactivity detector with the same structure. Time is simulated, so the output is deterministic.

Java Stand-in: inactivity alert with state and timers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Device d2 spoke only once, so its timer was due at t=7 and fires as soon as the clock moves on (shown at t=8). Device d1 kept pushing its timer ahead; it alerts only at the final clock advance, after the stream ended.

Another classic: deduplication with state TTL

Emit only the first event for each ID; ignore later ones. The state keeps a flag per ID.

Java
public class FirstOnly extends KeyedProcessFunction<String, Event, Event> {
    private transient ValueState<Boolean> seen;

    @Override
    public void open(OpenContext c) {
        StateTtlConfig ttl = StateTtlConfig.newBuilder(Duration.ofHours(1)).build();   // expire after 1 h
        ValueStateDescriptor<Boolean> d = new ValueStateDescriptor<>("seen", Types.BOOLEAN);
        d.enableTimeToLive(ttl);
        seen = getRuntimeContext().getState(d);
    }

    @Override
    public void processElement(Event e, Context ctx, Collector<Event> out) throws Exception {
        if (seen.value() == null) {
            seen.update(true);
            out.collect(e);
        }
    }
}

State TTL removes the flag an hour after its last write, so the state does not grow without bound.

Java Stand-in: deduplicate events by ID

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

When to use what

Need Tool
Transform or filter records map, filter, flatMap
Running aggregate per key reduce, sum, ...
Aggregate over time ranges Windows
Custom state, timers, side outputs, multi-step logic KeyedProcessFunction

Process functions are powerful but low level: you own state cleanup and timer logic. Reach for them when a higher-level operator would be a contortion.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which function type gives you keyed state AND timers?

    State and timers need a key, so they are available in KeyedProcessFunction on a KeyedStream.
  2. 2. When does onTimer run, and for which key?

    Timers are scoped to keys; state in onTimer refers to that key.
  3. 3. What triggers an event-time timer?

    Event-time timers follow watermarks, not the wall clock.
  4. 4. Why is state TTL useful in a deduplication function?

    Without expiry, every ID ever seen stays in state forever.

Summary

  • Process functions combine records, keyed state, timers and side outputs; KeyedProcessFunction is the workhorse.
  • processElement handles records; onTimer handles timers registered earlier, with the same key.
  • Timers are checkpointed and deduplicated; event-time timers fire on watermarks.
  • Always plan state cleanup: delete timers, clear state or use TTL.

Next: Connect, Union and Side Outputs.