Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Time and Windows

Watermarks

Learn how Flink watermarks measure progress of event time, how forBoundedOutOfOrderness works, how watermarks combine across inputs, and how to handle idle sources.

If events can arrive out of order, how does Flink know that "the 10:00 minute is complete" and a window can safely produce its result? It cannot see the future. What it can do is estimate how far event time has progressed, and promise that no (or almost no) earlier events will come. That promise is a watermark.

Watermarks are the heart of event-time processing, and the concept that confuses people most. This lesson builds the idea up step by step. It follows Event Time and Processing Time.

What a watermark means

A watermark with value W(t) is a special record in the stream that says: "I believe all events with timestamp up to t have now arrived." Operators react to it:

  • An event-time window ending at time end fires when a watermark with t >= end - 1 ms arrives, because it can now assume the window is complete.
  • An event-time timer at time T fires when the watermark reaches T.
  • Records with a timestamp at or below the current watermark are considered late: the system had already promised not to see them. (How to deal with them is the topic of Late Data and Allowed Lateness.)

Watermarks are not event data; they flow along the stream as markers. A watermark is also a trade-off dial: if you declare progress too early you drop or mishandle late data; if too late, results are delayed.

Bounded out-of-orderness

The standard strategy assumes events can be late by at most a fixed amount. The watermark is then

Text
watermark = (largest timestamp seen so far) - maxOutOfOrderness - 1 ms
Java
WatermarkStrategy<Reading> strategy = WatermarkStrategy
    .<Reading>forBoundedOutOfOrderness(Duration.ofSeconds(5))
    .withTimestampAssigner((r, recordTs) -> r.timestamp());

With 5 seconds of allowed disorder, when the largest timestamp seen is 10:00:42, the watermark is 10:00:37 (less 1 ms): events from before that are assumed complete. If an event stamped 10:00:39 shows up after the maximum reached 10:00:42, it is still not late (39 is above the watermark of 37). One stamped 10:00:35 is late.

For perfectly ordered data, WatermarkStrategy.forMonotonousTimestamps() is forBoundedOutOfOrderness with zero delay.

Stand-in: watermark progress and lateness

Java Stand-in: a bounded-out-of-orderness watermark generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Event c (ts 9) arrives after b (12) but is not late: the watermark was still low enough. Event e (ts 14) arrives after d (ts 20) has pushed the watermark to 14, so e counts as late. Notice also that the watermark only increases.

When are watermarks generated?

The generator emits watermarks periodically, by default every 200 ms (env.getConfig().setAutoWatermarkInterval(ms)), based on the largest timestamp seen. A custom WatermarkGenerator can also emit them per event (for example when a special end-of-batch marker appears):

Java
public class PunctuatedGenerator implements WatermarkGenerator<Event> {
    @Override
    public void onEvent(Event e, long eventTimestamp, WatermarkOutput out) {
        if (e.isEndOfBatch()) {
            out.emitWatermark(new Watermark(eventTimestamp));
        }
    }
    @Override
    public void onPeriodicEmit(WatermarkOutput out) { }
}

Generating watermarks as close to the source as possible (inside the Kafka source, not after a map) is best practice. With a partitioned source, each partition tracks its own watermark, so a lagging partition correctly holds back the combined watermark.

Watermarks through the pipeline

An operator with several input channels (after a keyBy, for example) receives watermarks from each of them. Its current watermark is the minimum of all its inputs' watermarks. The slowest input rules, because the operator can only promise what every input has promised.

flowchart LR
    A["Partition A<br/>watermark 100"] --> W["Window operator<br/>watermark = min = 60"]
    B["Partition B<br/>watermark 60"] --> W
    C["Partition C<br/>watermark 120"] --> W

The operator's event-time clock is the minimum of its inputs' watermarks.

Java Stand-in: combining watermarks from several inputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Until every input has reported, the operator has no watermark at all. The result never moves forward faster than the slowest input.

The idle source problem

If one input produces no events (a quiet Kafka partition at night, an unused shard), it also produces no watermarks, so the minimum stays frozen and no window ever fires, even though other inputs keep sending data. The fix is idleness detection:

Java
WatermarkStrategy.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(5))
    .withTimestampAssigner((e, ts) -> e.timestamp())
    .withIdleness(Duration.ofMinutes(1));      // after 1 minute of silence, ignore this input

An idle input is excluded from the minimum until it produces data again. Another common cause of stuck windows is fewer Kafka partitions than the source's parallelism: the subtasks without a partition never advance. Set the parallelism no higher than the partition count, or use idleness.

Java Stand-in: a silent input blocks the watermark until marked idle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Choosing the delay

  • Measure the real lateness of your data (the gap between event time and arrival) and pick a delay that covers most events, for example the 99th percentile.
  • A bigger delay means more complete results but results come later and state is held longer.
  • A smaller delay is faster, with more late events you must handle with allowed lateness and side outputs.
  • The window fires at end + delay: a 1-minute window with 10 s of disorder produces its result about 10 s after the minute ends.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a watermark W(t) assert?

    Watermarks measure event-time progress and trigger windows and timers.
  2. 2. With forBoundedOutOfOrderness(5 s), the largest timestamp seen is 100 s. What is the watermark (approximately)?

    Watermark = max timestamp - allowed disorder - 1 ms, about 95 s.
  3. 3. An operator receives watermarks from three inputs: 100, 60 and 120. Its watermark is:

    The minimum of all inputs, because every input must have promised.
  4. 4. What problem does withIdleness() solve?

    Idle inputs are excluded from the minimum until they send data again.

Summary

  • A watermark is a promise about event-time progress; windows and event-time timers fire when it passes their time.
  • forBoundedOutOfOrderness(d) computes max timestamp - d - 1 ms; watermarks never go backwards and are generated periodically (200 ms by default).
  • Operators use the minimum watermark across their inputs, so one slow or silent input stalls everything: use withIdleness and match parallelism to partitions.
  • The delay is a trade-off between completeness and latency.

Next: Tumbling and Sliding Windows.