Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 48 ยท Streaming

Event Time, Windows and Watermarks

Group streaming events by when they happened: tumbling and sliding windows, late data, watermarks, state cleanup and exactly-once output, with a simulator.

A streaming job usually asks time-based questions: how many clicks per minute, what was the average temperature in each five-minute span. The tricky part is which clock you mean. A phone may record a tap at 12:00:03, lose signal in a tunnel, and deliver the event at 12:04:40. Counting that tap in the minute it arrived would be wrong; it belongs in the minute it happened.

This lesson covers event time (when the event happened), windows that group events by event time, and watermarks, which tell Spark how long to wait for stragglers before closing a window and freeing its memory. You should know the basics from Structured Streaming Basics.

Event time versus processing time

  • Event time is a timestamp stored inside the record, set by the device or service that produced it.
  • Processing time is the clock of the Spark cluster when it handles the record.

Events can arrive late and out of order, so processing time gives results that change with network luck. Structured Streaming lets you aggregate by event time, which gives the same answer however the events are delayed. The cost is that Spark must keep the partial result for each window until it is sure no more events for it will come. That decision is the watermark.

Windows

A window is a time interval. Grouping by a window is an ordinary groupBy:

Python
from pyspark.sql.functions import window, col

counts = (events
          .groupBy(window(col("event_time"), "10 minutes"), col("page"))
          .count())

window(col, "10 minutes") creates a struct column with start and end. This is the tumbling window: fixed size, no overlap, every event belongs to exactly one window.

A sliding window adds a slide interval smaller than the window size:

Python
window(col("event_time"), "10 minutes", "5 minutes")

Windows are 10 minutes long but a new one starts every 5, so each event falls into two windows. Sliding windows suit moving averages ("the last 10 minutes, refreshed every 5").

flowchart TB
    subgraph tumbling["Tumbling, size 10"]
      A1["0 to 10"]
      A2["10 to 20"]
      A3["20 to 30"]
    end
    subgraph sliding["Sliding, size 10, slide 5"]
      B1["0 to 10"]
      B2["5 to 15"]
      B3["10 to 20"]
      B4["15 to 25"]
    end

Tumbling windows tile the timeline; sliding windows overlap, so one event lands in several of them.

There is also a session window (session_window(col, "5 minutes")) whose length depends on activity: it stays open as long as events keep arriving within the gap.

The stand-in assigns events to windows the same way Spark does: a window starts at a multiple of the slide.

Python Assigning events to tumbling and sliding windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Late data and the watermark

Spark must keep a window's running total in state while the window can still receive events. Without a limit, state would grow forever, because someone, somewhere, might send an event from three days ago.

A watermark sets that limit. You declare how late data can be:

Python
counts = (events
          .withWatermark("event_time", "10 minutes")
          .groupBy(window(col("event_time"), "5 minutes"), col("page"))
          .count())

Spark tracks the largest event time seen so far and computes:

watermark = max event time seen - 10 minutes

The watermark is calculated at the end of each micro-batch and applies from the next one. Its rules:

  • Events older than the watermark may be dropped. Spark guarantees to process data that is less than the delay behind, but it does not promise to drop everything older (an old event might still be processed if its window state still exists).
  • Windows whose end is at or before the watermark are complete. Their state is removed, freeing memory.
  • The watermark only moves forward. An old event never moves it back.

The delay is a trade-off. A long delay accepts very late events but keeps more state and delays final results; a short delay is cheap but drops more late events. Pick it from how late your data really is (measure it).

Watermarks and output modes

  • append with a windowed aggregation emits a window once, when the watermark passes its end. Results appear later, but they are final.
  • update emits updated counts as events arrive; a window may be written several times. The sink must handle updates.
  • complete ignores the watermark for output (it never drops old windows), so state grows without bound. Avoid it for time windows on long streams.

Aggregations without a watermark cannot run in append mode, because Spark could never be sure a window is finished.

The next stand-in runs a watermark by hand with a 5-second delay. Four micro-batches arrive with some late events. Watch which events are accepted, which are dropped, and when windows are emitted.

Python A watermark in action (tumbling windows of 10 s, delay 5 s)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this run the late 8 was counted in the first window, while 4 and 6 were dropped. Window [0, 10) waited until the watermark reached 10 before being emitted with its final count.

State and fault tolerance

Window counts live in Spark's state store, kept in executor memory and backed up to the checkpoint directory after every batch. If the application restarts, state is restored from the checkpoint, so a half-finished window continues counting. The default store is a hash map in memory; the RocksDB state store (spark.sql.streaming.stateStore.providerClass) keeps state on local disk and is the better choice when state holds millions of keys.

Remember that state is tied to the query's shape. Changing the window size, the grouping columns or the aggregate functions while reusing the checkpoint usually fails with a state schema error. Use a new checkpoint folder and reprocess if you must change them.

Exactly-once results

Structured Streaming aims for end-to-end exactly-once processing. Three ingredients work together:

  1. A replayable source with offsets (files, Kafka) so a failed batch can be read again.
  2. Checkpointed offsets and state so Spark knows which batches are done.
  3. An idempotent sink, one where writing the same batch twice leaves the same result. The file sink does this with a metadata log; for databases you make it so with upserts keyed by something like the window and the batch id.

Without all three you get at-least-once, meaning duplicates after a failure. Writing to a non-idempotent sink with foreach is the usual way to lose the guarantee; Streaming with Kafka and Delta shows how to keep it.

Real versus stand-in. Real Spark computes the watermark per micro-batch from the maximum event time across all partitions, stores state in a versioned state store and cleans it up after each commit. Dropping of late rows in Spark is best-effort, not a hard cutoff. The stand-in applies a strict cutoff in a few lines to make the rules easy to follow.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why aggregate by event time instead of processing time?

    Event time is stored in the record, so a delayed event still lands in the window where it belongs.
  2. 2. A 10-minute window slides every 5 minutes. In how many windows does one event fall?

    Window size divided by slide gives the number of overlapping windows, here 10 / 5 = 2.
  3. 3. The maximum event time seen is 12:30 and the watermark delay is 10 minutes. What is the watermark?

    watermark = max event time seen minus the delay = 12:20. Windows ending at or before it are complete.
  4. 4. Why can a windowed aggregation not use append mode without a watermark?

    Append mode writes rows that will never change. Only the watermark tells Spark that no more events can update a window.
  5. 5. What must the sink be for end-to-end exactly-once results?

    After a failure Spark may re-run a batch. A replayable source plus checkpoints plus an idempotent sink turns that into exactly-once output.

Summary

  • Event time is the timestamp inside the record; windows group by it with window(col, size[, slide]).
  • Tumbling windows do not overlap; sliding windows do; session windows follow activity gaps.
  • A watermark (withWatermark) equals the maximum event time seen minus the allowed delay; it closes windows, frees state and bounds lateness.
  • In append mode a window is emitted once the watermark passes its end; update emits changes; complete keeps all state.
  • State lives in a state store backed up to the checkpoint, and it is tied to the shape of the query.
  • Exactly-once output needs a replayable source, a checkpoint and an idempotent sink.

Next: connect streams to real message brokers and tables in Streaming with Kafka and Delta.