Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 48 ยท Streaming

Streaming with Kafka and Delta

Read from and write to Kafka with Structured Streaming, manage offsets, use foreachBatch with idempotent writes, and learn how Delta Lake logs commits.

Real streaming pipelines rarely read from a folder of files. Events usually flow through a message broker, most often Apache Kafka, and the results land in a table that other jobs and dashboards can query. Spark's most common combination for this is Kafka as the source and Delta Lake (or another table format) as the sink.

This lesson shows how a Kafka source and sink look in Structured Streaming, how offsets are tracked, how to run custom code per micro-batch with foreachBatch, and why idempotent writes matter. It then introduces Delta Lake's transaction log. Kafka, Spark and Delta cannot run on this site, so the real code is read-only and the TryIts are stand-ins that reproduce the logic. Review Structured Streaming Basics and Event Time, Windows and Watermarks first.

Kafka in sixty seconds

Kafka stores events in topics. A topic is split into partitions, and each partition is an append-only log in which every record has an increasing number, its offset. Producers append records; consumers read from an offset they choose and move forward. Kafka does not delete a record when it is read, so many readers can process the same data and a reader can go back in time (until the retention period removes old records).

flowchart LR
    P["Producers"] --> T0["Topic orders<br/>partition 0<br/>offsets 0..N"]
    P --> T1["Topic orders<br/>partition 1<br/>offsets 0..M"]
    T0 --> S["Spark readStream<br/>one task per partition range"]
    T1 --> S
    S --> D[("Delta table")]

Spark reads each partition from the offset after the last batch, so a replay after a failure covers exactly the same range.

The Kafka source

The connector is a separate package (spark-sql-kafka-0-10), added with --packages on spark-submit. The code below is real PySpark, read-only:

Python
from pyspark.sql.functions import col, from_json
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

raw = (spark.readStream
       .format("kafka")
       .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092")
       .option("subscribe", "orders")
       .option("startingOffsets", "earliest")
       .option("maxOffsetsPerTrigger", 100000)
       .load())

schema = StructType([
    StructField("order_id", StringType()),
    StructField("country", StringType()),
    StructField("amount", DoubleType()),
    StructField("event_time", TimestampType()),
])

orders = (raw
          .select(from_json(col("value").cast("string"), schema).alias("o"),
                  col("topic"), col("partition"), col("offset"))
          .select("o.*", "topic", "partition", "offset"))

The Kafka source always produces the same fixed columns: key and value (both binary), topic, partition, offset, timestamp and timestampType. You cast value to a string and parse it, usually with from_json (see Arrays, Maps, Structs and JSON) or an Avro/Protobuf decoder.

Useful options:

Option Meaning
subscribe, subscribePattern, assign Which topics or partitions to read
startingOffsets earliest, latest or a JSON of exact offsets. Used only on the first run; afterwards the checkpoint decides. Streaming default is latest.
maxOffsetsPerTrigger Caps rows per micro-batch, protecting you from a huge backlog
failOnDataLoss Fail (default) or continue if requested offsets were already deleted by retention
minPartitions Split big Kafka partitions into more Spark tasks

Who tracks the offsets?

Spark does. It stores the offset ranges of each batch in the checkpoint directory and does not rely on Kafka's consumer-group commits. This is why checkpointLocation is mandatory for a reliable query. If you delete the checkpoint, the query forgets its position and uses startingOffsets again.

The Kafka sink

To write to Kafka, produce a DataFrame with a value column (string or binary), optionally key, and optionally topic:

Python
from pyspark.sql.functions import to_json, struct

(orders
 .select(col("order_id").alias("key"),
         to_json(struct("order_id", "country", "amount")).alias("value"))
 .writeStream
 .format("kafka")
 .option("kafka.bootstrap.servers", "broker1:9092")
 .option("topic", "orders-enriched")
 .option("checkpointLocation", "/chk/orders-enriched")
 .start())

The Kafka sink is at-least-once: after a failure a batch may be written again, so consumers must tolerate duplicates (deduplicate by key) or use Kafka transactions on their side.

foreachBatch: custom logic per micro-batch

Not every destination has a streaming sink. foreachBatch gives you each micro-batch as a normal DataFrame plus a batch id, so you can use any batch writer (JDBC, Delta MERGE, multiple outputs):

Python
def save(batch_df, batch_id):
    (batch_df.write
        .format("jdbc")
        .option("url", "jdbc:postgresql://db/shop")
        .option("dbtable", "orders_staging")
        .mode("append")
        .save())

orders.writeStream.foreachBatch(save).option("checkpointLocation", "/chk/pg").start()

Be careful: if the application crashes after save finished but before Spark committed the batch, Spark runs save again for the same batch_id with the same data. A naive append then inserts duplicates.

Idempotent writes

An operation is idempotent when doing it twice has the same effect as doing it once. Two standard ways to get there:

  • Upsert by a key. Use MERGE, INSERT ... ON CONFLICT DO UPDATE, or a key-value store, so a repeated row replaces itself.
  • Record the batch id. Write the batch id with the data, or in a separate table in the same transaction, and skip a batch whose id was already applied.

The stand-in below simulates a Kafka partition, a consumer that crashes between writing and committing, and two sinks: a naive append and an idempotent upsert. The replay duplicates rows only in the naive version.

Python Replay after a crash: append versus idempotent upsert

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Delta Lake basics

Plain Parquet folders are fragile for streaming: concurrent writers can leave half-written files, readers can see partial results, and you cannot update a row. Delta Lake is an open table format that adds a transaction log on top of Parquet files. It is a library (delta-spark) plus a configuration, not a separate server:

Python
from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
         .config("spark.sql.catalog.spark_catalog",
                 "org.apache.spark.sql.delta.catalog.DeltaCatalog")
         .getOrCreate())

(orders.writeStream
       .format("delta")
       .option("checkpointLocation", "/chk/orders-delta")
       .outputMode("append")
       .start("/lake/orders"))

spark.read.format("delta").load("/lake/orders").show()
spark.read.format("delta").option("versionAsOf", 3).load("/lake/orders")   # time travel

Inside the table folder, next to the Parquet data files, sits a _delta_log directory. Every write is a commit: a small JSON file (00000000000000000000.json, 00000000000000000001.json, and so on) listing which data files were added or removed. The current state of the table is found by replaying the log. Because a commit becomes visible only when its log file is written atomically, readers see a whole commit or none of it. This gives:

  • ACID transactions for batch and streaming writers on the same table.
  • Time travel: read an older version, or restore it.
  • MERGE, UPDATE, DELETE: rewriting only the affected files.
  • Schema enforcement and controlled schema evolution.
  • Idempotent streaming writes: the Delta sink stores the query id and batch id in the log and skips a batch it has already committed.

Delta is not the only table format; Apache Iceberg and Apache Hudi solve the same problems with different designs.

The next stand-in is a toy transaction log. Each commit records added and removed files. Replaying the first N commits gives the table state at version N-1, which is the idea behind time travel.

Python A toy transaction log with time travel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real Kafka distributes partitions across brokers with replication, and the Spark connector plans offset ranges per micro-batch, then reads them with Kafka consumers inside executor tasks. Real Delta stores commits as JSON files with periodic Parquet checkpoints and uses optimistic concurrency control. The stand-ins use lists and dicts to reproduce only the logic: offsets give replay, idempotent writes absorb replays, and a log of file changes gives versions.

Production checklist

  • Give every query its own checkpoint folder on reliable storage.
  • Set maxOffsetsPerTrigger so a restart after downtime does not try to read a whole day in one batch.
  • Make every sink idempotent (upsert, batch-id guard, Delta).
  • Watch lastProgress: input rate versus processing rate, and offset lag in Kafka.
  • Plan for small files: frequent triggers write many small Delta files, so schedule compaction (OPTIMIZE) and clean-up (VACUUM).
  • Test failure recovery by killing the job on purpose.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Who stores the progress (offsets) of a Structured Streaming query reading Kafka?

    Spark writes offset ranges to its checkpoint before and after each batch, and uses them on restart. startingOffsets matters only for the first run.
  2. 2. In what form do the key and value columns arrive from the Kafka source?

    The source exposes raw bytes. Cast value to string and then use from_json, or decode Avro or Protobuf.
  3. 3. A foreachBatch function appends rows to a database. Why can that produce duplicates?

    Spark guarantees at-least-once for custom code. Use upserts or record the batch id to make the write idempotent.
  4. 4. What makes a Delta Lake table's commits atomic?

    Readers learn which files belong to the table from the _delta_log. A commit is a log entry listing added and removed files, so it appears all at once.

Summary

  • Kafka topics are partitioned logs; each record has an offset, and Spark reads exact offset ranges per micro-batch.
  • The Kafka source returns binary key and value plus topic, partition, offset and timestamp; cast and parse them.
  • Spark tracks offsets in its own checkpoint; startingOffsets applies only to the first run; maxOffsetsPerTrigger limits batch size.
  • The Kafka sink is at-least-once, so consumers should tolerate duplicates.
  • foreachBatch runs batch code on each micro-batch; make it idempotent with upserts or batch ids.
  • Delta Lake adds an atomic transaction log to Parquet, giving ACID writes, time travel, MERGE and idempotent streaming sinks.

Next: leave streaming and learn Spark's machine-learning library in MLlib Overview and Pipelines.