Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 ยท Streaming

Watermarks and Late Data

Learn how watermarks track event-time progress, decide when windows close, and how allowed lateness, side outputs and corrections handle late events.

When you window by event time, a window for 09:00-09:05 cannot be closed at 09:05 on the clock, because events stamped 09:04 might still be on their way. Wait forever and you never produce a result; close too early and you drop data. A watermark is the stream processor's estimate of how far event time has progressed. It answers: "I believe no more events older than T will arrive." This lesson shows how that estimate is computed and what to do with events that violate it.

What a watermark means

A watermark with value W asserts that all events with event time <= W have been seen. When the watermark passes the end of a window, the window is complete and its result can be emitted.

A common strategy is bounded out-of-orderness: the watermark is the largest event time seen so far minus a fixed delay.

Text
watermark = max_event_time_seen - allowed_delay

A bigger delay waits longer (more complete results, more latency); a smaller delay is faster but risks more late events.

Python Watermark by bounded delay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The event at time 2 arrives after the watermark has reached 7, so it is late. The watermark never moves backward.

Closing windows with a watermark

Combine tumbling windows with the watermark: keep each window's running count in state; when the watermark passes a window's end, emit and discard it.

Python Emit windows when the watermark passes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Event 8 arrives before the watermark passes the end of [0,10), so it still counts. Event 3 arrives after that window has closed, so it is dropped. The result trades completeness for timeliness, and the delay is the knob.

Handling late events

When an event arrives behind the watermark you have options:

Late data options

Drop Simple, but loses data.
Allowed lateness Keep windows open longer.
Side output Send late events elsewhere.
Correct later Batch reprocess history.

Always count what you drop.

1 / 2
  1. Drop and count them. Acceptable only if you measure how much is lost.
  2. Allowed lateness: keep a closed window's state for an extra period; if a late event arrives within it, emit an updated result. Consumers must then accept corrections (upserts) rather than treat the first output as final.
  3. Side output: route late events to a separate stream or table for inspection or batch repair.
  4. Reconcile with batch: run a nightly batch recomputation over event-time partitions, which overwrites the approximate streaming results. This is the idea behind the Lambda pattern from Pipeline Architecture Patterns.
Python Allowed lateness with updates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The late event at time 6 changes window [0,10) from 2 to 3, emitted as an update. The event at time 2 arrives after the lateness period and goes to the side output. Downstream tables must therefore upsert by window, not append.

Idle sources stall the watermark

If one partition goes quiet, the overall watermark (the minimum across partitions) stops advancing and no windows close. Real engines handle this with an idle timeout that excludes silent partitions from the minimum. When a stream job seems stuck with no output, check for an idle partition first.

Python Watermark as the minimum across partitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing the delay

Measure, do not guess. Look at the distribution of lateness (arrival time minus event time) in recent data, pick a delay that covers most events (say the 99th percentile), and track the dropped-event count as a metric. Mobile apps that batch uploads need minutes or hours; server logs shipped directly need seconds.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a watermark of 100 assert?

    A watermark is an estimate that event time has progressed past that point, so earlier windows can close.
  2. 2. With bounded out-of-orderness, how is the watermark computed?

    The watermark trails the newest event time by the allowed delay.
  3. 3. What must downstream tables support if windows can be updated after first emission?

    Corrections replace earlier results, so consumers need to upsert.
  4. 4. A stream job stops emitting windows although events flow. What is a common cause?

    The global watermark is the minimum across partitions, so a silent partition stalls it unless idle detection is configured.

Summary

  • A watermark estimates event-time progress and tells the engine when a window can close.
  • Bounded out-of-orderness: watermark = max event time seen minus a delay; bigger delay means more completeness and more latency.
  • Late events can be dropped (and counted), kept via allowed lateness with updates, routed to a side output or repaired by batch.
  • Idle partitions can stall watermarks; pick delays from measured lateness.

Next: Delivery Guarantees and Deduplication