Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 36 ยท Streaming

Event Time and Windows

Learn the difference between event time and processing time, and compute tumbling, sliding and session windows over a stream with runnable Python code.

A stream never ends, so "the total number of clicks" has no final answer. Useful questions are about windows: clicks per minute, orders in the last hour, sessions of activity. To define a window you need a clock, and in streaming there are two clocks that disagree. Understanding the difference is the foundation of correct stream processing.

Two kinds of time

  • Event time is when the event actually happened, recorded by the source ("the phone logged the tap at 09:00:03").
  • Processing time is when your system sees the event ("it arrived at 09:00:41").

If networks were instant and ordered they would match. In reality, phones go through tunnels, batches are uploaded later and partitions are read at different speeds. Events arrive late and out of order. If you window by processing time, a late event is counted in the wrong window and results depend on how busy the system was. If you window by event time, results reflect reality but you must decide how long to wait for stragglers (next lesson).

Python Out of order arrival

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Events d and f arrived after events with larger timestamps. A correct windowed count must still place them in their event-time windows.

Tumbling windows

A tumbling window has a fixed size and does not overlap: 00:00-00:05, 00:05-00:10, and so on. Every event belongs to exactly one window, found by integer division of the timestamp.

Python Tumbling window counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The window start is t // size * size. Use tumbling windows for "per minute" or "per hour" metrics.

Sliding (hopping) windows

A sliding window has a size and a smaller slide; windows overlap, so one event belongs to several windows. "Events in the last 60 seconds, updated every 10 seconds" is a sliding window.

Python Sliding window counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each event now appears in three windows (size 30 divided by slide 10), so state and output are larger. Use sliding windows for moving averages and smoothing.

Session windows

A session window groups events of one key separated by less than a gap; a longer silence closes the session. Session length varies, which suits "user visits". They cannot be computed by integer division; you group by gaps.

Python Session windows with a gap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a window

Tumbling Fixed, no overlap.
Sliding Fixed size, overlap.
Session Closed by inactivity.

Event time decides the window, not arrival.

1 / 2

Windowing in SQL

Batch SQL can window by flooring a timestamp, and stream engines expose the same idea through functions such as TUMBLE, HOP and SESSION. The SQL below buckets events into minutes, which is the tumbling window in batch form:

Python Tumbling window in SQL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Minute 09:02 is absent because no events fell into it. Real stream engines emit a window result when it closes, and empty windows usually produce no row unless you ask for them.

Aggregations that windows need

Windows keep state until they are closed. Sums, counts, minimum and maximum are cheap to keep (one number). Exact distinct counts and medians need much more memory, so engines offer approximations (HyperLogLog sketches, t-digests). Pick the aggregation with the state size in mind.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is event time?

    Event time is recorded by the source; processing time is when your system sees it.
  2. 2. Which window type assigns every event to exactly one fixed-size window?

    Tumbling windows are fixed and non-overlapping, so each event falls in exactly one.
  3. 3. A 60-second window that advances every 10 seconds is:

    Windows overlap when the slide is smaller than the size, so this is a sliding window.
  4. 4. What closes a session window?

    Session windows are defined by activity, closing after a quiet period.

Summary

  • Event time is when it happened; processing time is when you saw it; they differ because of delay and reordering.
  • Tumbling windows are fixed and disjoint, sliding windows overlap, session windows close after inactivity.
  • Windows hold state until they close, so aggregation choice affects memory.
  • Windowing by event time requires deciding how long to wait for late events.

Next: Watermarks and Late Data