Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 48 ยท Streaming

Structured Streaming Basics

Learn the Structured Streaming model: an unbounded table, sources, sinks, triggers, output modes and checkpoints, with a runnable micro-batch simulator.

Batch jobs answer questions about data that already exists: yesterday's orders, last month's logs. Streaming jobs answer them while the data is still arriving: orders per minute, errors in the last five minutes, a live fraud score. Structured Streaming is Spark's streaming engine, and its central promise is that you write a streaming query almost exactly like a batch query. Spark then runs it again and again as new data shows up.

This lesson explains the mental model (an unbounded table), the pieces of every streaming query (source, transformation, sink, trigger, output mode, checkpoint), and ends with a simulator you can run to see micro-batches in action. You should know DataFrames from DataFrames: Creating and Inspecting and aggregations from the DataFrame lessons.

The unbounded table model

Imagine a table that never stops growing. New rows are appended at the bottom as events arrive. A streaming query is an ordinary query over that table: select, filter, groupBy, join. Spark does not recompute the whole thing for every new row. It works out the incremental change and updates the result.

flowchart LR
    S["Input stream<br/>rows keep arriving"] --> T["Unbounded input table"]
    T --> Q["Query<br/>filter, groupBy, count"]
    Q --> R["Result table"]
    R --> K["Sink<br/>files, Kafka, console"]

Every trigger appends new rows to the input table, the query updates the result table, and the sink receives the change.

Because the code looks like batch code, you can often test a streaming query on a static DataFrame first and switch to readStream later. Only a few operations are unsupported on streams (for example a global sort without aggregation, or limit and distinct in most cases), because they would need the entire unbounded input.

By default Spark executes the query as micro-batches: every so often it collects the newly arrived rows, runs a normal Spark job on them, and commits the progress. Latency is typically around a second or a few hundred milliseconds. There is also an experimental continuous mode for millisecond latency, rarely used.

A first streaming query

This is real PySpark. The source is a folder of JSON files and the sink is the console. Both are shown read-only because the site cannot run Spark:

Python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("orders-stream").getOrCreate()

orders = (spark.readStream
          .format("json")
          .schema("order_id STRING, country STRING, amount DOUBLE")
          .option("maxFilesPerTrigger", 1)
          .load("/data/incoming/orders/"))

big = orders.filter(col("amount") > 100)

query = (big.writeStream
         .format("console")
         .outputMode("append")
         .option("checkpointLocation", "/tmp/checkpoints/big-orders")
         .start())

query.awaitTermination()

Look at the three parts:

  • spark.readStream returns a streaming DataFrame. It looks like any other DataFrame, but df.isStreaming is True.
  • The transformations are the same ones you already know.
  • writeStream ... start() launches the query in the background and returns a StreamingQuery handle. awaitTermination() keeps your program alive; without it the script would end and take the query with it.

File sources need an explicit schema (or spark.sql.streaming.schemaInference=true), because Spark does not want to scan an infinite input to guess types.

Sources

A source is where rows come from. Built-in ones:

Source Use
File (json, csv, parquet, orc, text) Watches a directory for new files. Files must appear atomically (write elsewhere, then move in).
Kafka The standard choice for real systems; see Streaming with Kafka and Delta.
Socket Reads text from a TCP port. For demos only, it is not fault tolerant.
Rate Generates timestamp and value rows at a fixed speed. Handy for tests and experiments.

Replayable sources (files, Kafka) remember offsets: positions that say how far the query has read. That is what makes recovery possible.

Sinks

A sink is where results go: file formats (append only), Kafka, console and memory (for debugging only, never production), and foreachBatch or foreach for custom code such as writing to a database. foreachBatch hands you each micro-batch as an ordinary DataFrame, so you can reuse batch writers.

Output modes

The output mode says which part of the result table is written at each trigger:

Mode Writes When it fits
append Only new rows that will never change Queries without aggregation (filter, map), or aggregations with a watermark whose windows are final
update Only rows that changed since the last trigger Running aggregates sent to a sink that can upsert
complete The whole result table every time Small aggregates, such as counts per country

Not every combination is legal. An aggregation without a watermark cannot use append, because a count may still grow. Spark checks this when you start the query and raises an AnalysisException.

Triggers

A trigger controls when a micro-batch starts:

Python
.trigger(processingTime="30 seconds")   # a batch every 30 seconds
.trigger(availableNow=True)             # process everything available, in batches, then stop
.trigger(once=True)                     # one batch, then stop (older; prefer availableNow)

With no trigger, the next batch begins as soon as the previous one finishes. availableNow=True is popular for incremental batch: a scheduled job that reads only the new files since last time (thanks to the checkpoint) and then ends, giving streaming semantics at batch cost.

Checkpoints and state

The checkpointLocation option points to a reliable folder (HDFS or object storage). Spark stores two things there:

  • Offsets and commit logs: which input range each batch covered and whether it finished.
  • State: for aggregations and joins, the intermediate data (for example the running counts).

If the application crashes or is stopped, restarting with the same checkpoint continues exactly where it left off. Without a checkpoint, a restart begins from scratch. Each query needs its own checkpoint folder, and you should not change the query's shape carelessly: adding a new aggregation or changing a grouping key may make the old state incompatible.

The first stand-in is a micro-batch engine. Each trigger takes the lines that arrived since the last one and updates a running word count. It then prints what complete and update would write.

Python A micro-batch word count with output modes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how update stays small while complete grows with the number of distinct words. With a billion distinct keys, complete would be impractical.

Watching and managing a query

The StreamingQuery object lets you inspect and stop what is running:

Python
query.status            # is it waiting for data, processing, or stopped?
query.lastProgress      # a dict: batch id, numInputRows, inputRowsPerSecond, durations
query.recentProgress    # the last several progress reports
query.exception()       # the error that stopped it, if any
query.stop()
spark.streams.active    # all running queries in this session

The Spark UI also has a Structured Streaming tab with input rate, processing rate and batch duration. The key health rule: processing time must stay below the trigger interval. If a batch takes 40 seconds and the trigger is every 30, you fall behind and latency grows without limit.

Recovering after a stop

The second stand-in imitates offsets and a checkpoint. The "source" is a numbered log of events. A query run processes whatever lies after the committed offset and then saves the new offset. We stop it, more data arrives, and a restart continues without repeating or skipping anything.

Python Offsets and restart from a checkpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Structured Streaming plans each micro-batch with the Catalyst optimizer, runs it as ordinary Spark jobs, and writes an offset log and commit log in the checkpoint directory before and after each batch, which together with an idempotent sink gives end-to-end exactly-once results. The stand-ins use a list, a counter and a dict to show the same sequence of ideas. Nothing here reads files, uses Kafka, or touches a cluster.

Common mistakes

  • No checkpoint in production. After a restart the query reprocesses or loses data.
  • memory or console sink in real jobs. They exist for debugging.
  • Sharing one checkpoint folder between queries. Each query needs its own.
  • Forgetting awaitTermination() in a script, so the program exits immediately.
  • Many tiny files. A short trigger on a file sink produces a small file per batch per partition; plan compaction (see Performance Tuning).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the central idea of Structured Streaming?

    Spark treats the stream as a table that keeps getting rows and runs your DataFrame query incrementally, usually as micro-batches.
  2. 2. A running count per country must be written, but only for countries whose count changed in the latest batch. Which output mode?

    update writes only the result rows that changed since the previous trigger. complete rewrites the whole table, and append is for rows that never change.
  3. 3. What does the checkpointLocation option store?

    The checkpoint holds the progress of the query (which input ranges were handled) and its aggregation or join state.
  4. 4. Which trigger runs everything currently available and then stops, making incremental batch jobs easy?

    availableNow=True processes all data that arrived since the checkpoint (in several batches if needed) and then terminates the query.

Summary

  • Structured Streaming treats a stream as an unbounded table and runs your batch-style query incrementally, normally as micro-batches.
  • readStream creates a streaming DataFrame, writeStream ... start() launches the query, and awaitTermination() keeps the program alive.
  • Sources include files, Kafka, socket and rate; sinks include files, Kafka, foreachBatch, console and memory (the last two for debugging).
  • Output modes are append, update and complete; legality depends on the query.
  • Triggers control batch timing; availableNow=True gives incremental batch.
  • A checkpoint stores offsets and state and is essential for recovery.

Next: handle late and out-of-order events with Event Time, Windows and Watermarks.