Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 48 ยท Streaming

Legacy Spark Streaming: DStreams

Understand the legacy DStream API: micro-batches, socketTextStream, windowed counts, updateStateByKey, checkpoints and why Structured Streaming replaced it.

Before Structured Streaming, Spark had a first streaming API called Spark Streaming, built on DStreams (discretized streams). You will still meet it in older code bases, tutorials and interview questions, even though it is deprecated for new work. This lesson explains how it works, shows its main operations (socketTextStream, window, updateStateByKey), and explains exactly why Structured Streaming replaced it, so you can read legacy code and migrate it.

Spark cannot run on this site, so the PySpark code below is read-only. The runnable examples are plain-Python stand-ins that simulate micro-batches.

The micro-batch idea

A DStream chops a live stream into small time slices, called batches, every few seconds (the batch interval). Each slice becomes an ordinary RDD, and Spark runs your transformations on each RDD like a tiny batch job. A DStream is therefore a sequence of RDDs over time.

flowchart LR
    S["Source<br/>(socket, Kafka)"] --> R["Receiver / direct reader"]
    R --> B1["RDD t0-t1"]
    R --> B2["RDD t1-t2"]
    R --> B3["RDD t2-t3"]
    B1 --> J1["Spark job"]
    B2 --> J2["Spark job"]
    B3 --> J3["Spark job"]

A DStream is a series of RDDs, one per batch interval; each RDD is processed by an ordinary Spark job.

The batch interval sets the lowest possible latency: with a 5-second interval, results arrive in multiples of 5 seconds. It also sets the load: if processing one batch takes longer than the interval, batches queue up and the application falls behind.

A word count over a socket

The classic first example reads lines from a TCP socket and counts words in each batch:

Python
from pyspark import SparkContext
from pyspark.streaming import StreamingContext

sc = SparkContext("local[2]", "DStreamWordCount")   # at least 2 threads: one receives, one processes
ssc = StreamingContext(sc, batchDuration=5)         # a 5-second batch interval

lines = ssc.socketTextStream("localhost", 9999)
counts = (lines.flatMap(lambda line: line.split())
               .map(lambda word: (word, 1))
               .reduceByKey(lambda a, b: a + b))
counts.pprint()                                     # an output operation, prints each batch

ssc.start()                                         # begin receiving and processing
ssc.awaitTermination()

The code reads like the RDD API because it is the same API applied to every batch. Nothing runs until ssc.start(); and, like batch Spark, transformations are lazy until an output operation (pprint, foreachRDD, saveAsTextFiles) is attached. Use nc -lk 9999 in a terminal to send text. Note that local[1] hangs: the receiver occupies the only thread, so nothing is processed.

Windows

A per-batch count only describes the last five seconds. To answer "words in the last minute, updated every 10 seconds" you use a window defined by a window length and a sliding interval. Both must be multiples of the batch interval.

Python
windowed = (lines.flatMap(lambda l: l.split())
                 .map(lambda w: (w, 1))
                 .reduceByKeyAndWindow(lambda a, b: a + b,       # add batches entering the window
                                       lambda a, b: a - b,       # subtract batches leaving it
                                       windowDuration=60,
                                       slideDuration=10))

The second function is the inverse reduce function. It lets Spark update the window incrementally (add the new batch, subtract the one that left) rather than re-adding all twelve batches each time. It requires checkpointing and an invertible operation such as sum. The simpler lines.window(60, 10) returns a DStream of the raw records inside the window.

Python A sliding window over micro-batches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

State across batches: updateStateByKey

Counting forever (a running total since the start) needs state that survives from one batch to the next. updateStateByKey keeps a state value per key and calls your function every batch with the new values and the old state:

Python
ssc.checkpoint("hdfs:///checkpoints/wordcount")           # required for stateful operations

def update(new_values, running_total):
    return sum(new_values) + (running_total or 0)          # return None to drop a key

totals = pairs.updateStateByKey(update)
totals.pprint()

Two costs are important. Checkpointing is mandatory, because lineage would otherwise grow without end. And updateStateByKey visits every key in the state on every batch, even keys with no new data, so cost grows with the total number of keys. The newer mapWithState (Scala/Java) only touches updated keys and supports timeouts, but PySpark never had it, which is one more reason people moved on.

Python updateStateByKey: a running total per word

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why Structured Streaming replaced DStreams

The DStream API works, but it has design limits that Structured Streaming was built to remove:

Topic DStreams Structured Streaming
API low-level RDDs DataFrames and SQL, same as batch
Time processing time only (batch arrival) event time, watermarks, late data handling
Optimizer none (RDD functions are opaque) Catalyst and Tungsten
Delivery guarantees at-least-once for most receivers; exactly-once needs care end-to-end exactly-once with idempotent sinks
State updateStateByKey visits all keys; manual checkpointing managed state store with timeouts
Reuse rewrite batch logic as RDD code write a DataFrame query once, run as batch or stream
Status maintenance mode, no new features the actively developed API

Event time is the biggest difference. DStreams group records by when they arrived, so a mobile event delayed by two minutes lands in the wrong window. Structured Streaming groups by the timestamp inside the record and tracks a watermark (see Event Time, Windows and Watermarks).

For comparison, the word count above in Structured Streaming is:

Python
lines = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()
counts = (lines.select(F.explode(F.split("value", " ")).alias("word"))
               .groupBy("word").count())
counts.writeStream.outputMode("complete").format("console").start().awaitTermination()

Real versus stand-in. Real DStreams run receivers on executors, replicate received blocks for reliability, and execute every batch as a distributed job with checkpointing. The stand-ins process Python lists in a single loop; they illustrate the batch-by-batch model and the window and state semantics, not fault tolerance or throughput.

If you inherit a DStream application, plan a migration: replace sources with readStream, express the logic as DataFrame operations, replace updateStateByKey with an aggregation or applyInPandasWithState/mapGroupsWithState, and test against the same input. See Streaming with Kafka and Delta for a modern source and sink.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a DStream?

    Spark Streaming discretizes the stream into time slices, each processed as an RDD.
  2. 2. Why does a DStream socket example fail to print anything with master local[1]?

    A receiver needs a dedicated core. Use at least local[2].
  3. 3. What is required to use updateStateByKey?

    State must be checkpointed so the lineage does not grow forever and the state can be recovered.
  4. 4. Which is a key reason Structured Streaming replaced DStreams?

    Event time with watermarks, Catalyst optimization and a unified batch/stream API are the main gains.

Summary

  • A DStream splits a live stream into micro-batches; each batch is an RDD processed by an ordinary Spark job.
  • StreamingContext(sc, interval), socketTextStream and an output operation such as pprint make a streaming job; nothing runs until ssc.start().
  • window and reduceByKeyAndWindow (with an optional inverse function) compute results over the last N seconds.
  • updateStateByKey keeps per-key state across batches, needs checkpointing and visits every key each batch.
  • DStreams are in maintenance mode: no event time, no optimizer, RDD-only. Use Structured Streaming for new code and migrate old code.

Next: bring pandas habits to big data in pandas API on Spark.