Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Streaming and Production

Structured Streaming Basics

Learn Structured Streaming: unbounded tables, readStream and writeStream, output modes, triggers, checkpoints, watermarks and windows, with runnable Python stand-ins.

Batch processing reads a fixed set of data, computes, and stops. Streaming handles data that keeps arriving: clicks, sensor readings, payments. Structured Streaming is Spark's streaming engine and Databricks' foundation for incremental pipelines. Its big idea is that a stream is just a table that keeps growing, and you write the same DataFrame code you would for a static table.

Databricks and Spark cannot run on this site, so Spark code is read-only. The runnable blocks are plain Python stand-ins that reproduce micro-batches, checkpoints and windows.

Read, transform, write

Python
from pyspark.sql import functions as F

events = (spark.readStream
          .table("prod.bronze.click_events"))          # a Delta table as a stream source

clicks_per_page = (events
    .where("event_type = 'click'")
    .groupBy("page")
    .count())

query = (clicks_per_page.writeStream
         .outputMode("complete")
         .option("checkpointLocation", "/Volumes/prod/sales/chk/clicks_per_page")
         .toTable("prod.gold.clicks_per_page"))

readStream instead of read, writeStream instead of write; everything between is ordinary DataFrame code. query is a handle: query.status, query.lastProgress, query.stop().

Sources include Delta tables, Auto Loader files (see Auto Loader), Kafka, Event Hubs, Kinesis and Pub/Sub. Sinks include Delta tables, Kafka and foreachBatch for custom code.

Micro-batches and triggers

Spark processes new data in small batches. The trigger says how often:

Trigger Behaviour
default (no trigger) Start the next batch as soon as the previous ends
processingTime="1 minute" One batch per minute
availableNow=True Process everything currently available, in several batches, then stop
realTime=... Low-latency mode on supported runtimes (check your version's docs)

availableNow is the star for cost control: it runs a streaming job as a scheduled batch job. You keep exactly-once, incremental behaviour without paying for a cluster that idles all day.

Python
(events.writeStream
   .trigger(availableNow=True)
   .option("checkpointLocation", chk)
   .toTable("prod.silver.events"))

Output modes

  • append: only new rows are written (use for non-aggregated results and for aggregates with a watermark).
  • update: only changed result rows are written.
  • complete: the whole result table is rewritten each time (small aggregates only).

Checkpoints: the memory of a stream

The checkpoint folder stores how far the query has read and the state of aggregations. If the job stops or crashes, it restarts from the checkpoint and continues without gaps or duplicates (for Delta sinks, exactly once). Rules:

  • One checkpoint location per query, never shared.
  • Do not delete it casually: removing it restarts the query from the beginning.
  • Changing certain query logic (the aggregation keys, sources) may require a new checkpoint.

The stand-in below simulates micro-batches with a checkpoint of "next offset" and shows a restart that does not reprocess data.

Python Stand-in: micro-batches with a checkpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Windows and watermarks

Aggregating "per 5 minutes" needs event time (when it happened) not arrival time. Late data is normal, but state cannot be kept forever, so you tell Spark how late is too late with a watermark.

Python
counts = (events
    .withWatermark("event_time", "10 minutes")
    .groupBy(F.window("event_time", "5 minutes"), "page")
    .count())

Windows older than (latest event time - 10 minutes) are finalised and dropped from state; events arriving even later are ignored. A longer watermark keeps more state and accepts later data.

Python Stand-in: tumbling windows with a watermark

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The late event at time 4 arrives when the newest event is 12, so with a 3-minute watermark it falls before the cutoff (9) and is dropped; the late event at 2 is dropped too.

Stateful operations and joins

Aggregations, deduplication (dropDuplicatesWithinWatermark), stream-stream joins and mapGroupsWithState/transformWithState keep state between batches. State lives in RocksDB on the workers and in the checkpoint. Always set watermarks on stateful operations, or state grows without limit. Stream-static joins (a stream joined to a Delta table) need no watermark.

Monitoring

  • query.lastProgress shows input rows, processing rate, batch duration and state size.
  • The Spark UI "Structured Streaming" tab graphs the same.
  • Alert when input rate and processed rate diverge: the stream is falling behind.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does trigger(availableNow=True) do?

    It turns an incremental streaming query into a finite, schedulable job.
  2. 2. What is the checkpoint used for?

    It records offsets and aggregation state.
  3. 3. Why set a watermark on a windowed aggregation?

    Without it, Spark must keep all windows forever.
  4. 4. Which output mode rewrites the whole result each batch?

    complete mode outputs the entire result table every time.

Summary

  • A stream is an ever-growing table; use readStream/writeStream with normal DataFrame code.
  • Triggers control batch frequency; availableNow gives cheap scheduled incremental runs.
  • Checkpoints provide fault tolerance and exactly-once writes to Delta.
  • Event-time windows need watermarks to bound state.

Next: Auto Loader