Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 39 ยท Data Formats and the Ecosystem

Kafka with Spark Structured Streaming and Flink

Read and write Kafka topics from Spark Structured Streaming and Flink: source and sink options, offsets and checkpoints, exactly-once sinks, with runnable stand-ins.

Kafka Streams runs inside your own application. Sometimes the job is bigger or lives in a different world: a data team that already runs Spark for batch analytics, or a pipeline that joins Kafka with files, databases and machine-learning steps. Then you use a separate processing engine and treat Kafka as its input and output. The two most common engines are Apache Spark (Structured Streaming) and Apache Flink.

Real versus stand-in. The Spark and Flink code below is real usage, read-only on this site because neither engine nor a Kafka cluster is available. The TryIt blocks are plain JavaScript stand-ins that imitate the key idea, checkpointed offsets, and are not Spark or Flink.

Why a separate engine?

A Streams application is a library: no cluster to manage, but it reads from Kafka and writes to Kafka. A processing engine runs as its own cluster and brings:

  • Many sources and sinks: Kafka, files, object storage, JDBC, lakehouse tables.
  • Large-scale state and SQL: both engines offer SQL on streams and handle very large state.
  • Unified batch and streaming (especially Spark) and rich libraries such as machine learning.

The price is another system to run. Choose it when the surrounding ecosystem demands it, as the Streams lesson explained.

Spark Structured Streaming: reading

Structured Streaming treats a stream as an unbounded table. You write DataFrame code as if it were a batch job, and Spark runs it incrementally. The Kafka source gives a DataFrame with fixed columns: key, value (both binary), topic, partition, offset, timestamp.

Python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, from_json, window
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

spark = SparkSession.builder.appName("orders").getOrCreate()

schema = StructType([
    StructField("customer", StringType()),
    StructField("total", DoubleType()),
])

orders = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "orders")
    .option("startingOffsets", "earliest")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("o"),
            col("timestamp"))
    .select("o.*", "timestamp"))

revenue = (orders
    .withWatermark("timestamp", "10 minutes")
    .groupBy(window("timestamp", "5 minutes"), "customer")
    .sum("total"))

The Kafka client options are passed with a kafka. prefix (kafka.security.protocol and so on). You choose topics with subscribe, subscribePattern or assign. startingOffsets applies only to the first run: once a checkpoint exists, the query resumes from it.

The package is separate from Spark itself. Add spark-sql-kafka-0-10 to the job (for example with --packages) and match its version to your Spark version.

Spark: writing

Python
query = (revenue
    .selectExpr("customer AS key", "to_json(struct(*)) AS value")
    .writeStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("topic", "revenue-per-customer")
    .option("checkpointLocation", "/data/checkpoints/revenue")
    .outputMode("update")
    .start())

The sink expects a value column (and optionally key, topic, headers). The checkpoint location is mandatory for reliable operation. It stores which offsets were processed and the aggregation state, on a durable file system or object store.

Offsets and checkpoints

Spark does not use consumer groups for progress. It reads a range of offsets per micro-batch, records those ranges in the checkpoint's write-ahead log, and only then runs the batch. After a crash it replays the same range. So group.id and committed offsets are not how a Spark job resumes, which surprises people looking at lag dashboards. The offsets Spark can optionally commit are informational.

Consequences:

  • Delete the checkpoint and the job starts over from startingOffsets.
  • Changing the query too much makes the old checkpoint incompatible.
  • If retention deletes offsets the checkpoint still points to, you get a data-loss error unless you set failOnDataLoss to false.
JS Why checkpointed offsets make restarts safe

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The range of the crashed batch is processed again after restart, but its first attempt never committed, so the result is counted once.

Flink has its own Kafka connector. The modern API is the KafkaSource and KafkaSink builders:

Java
KafkaSource<String> source = KafkaSource.<String>builder()
    .setBootstrapServers("localhost:9092")
    .setTopics("orders")
    .setGroupId("flink-orders")
    .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

DataStream<String> lines = env.fromSource(
    source,
    WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)),
    "orders-source");

KafkaSink<String> sink = KafkaSink.<String>builder()
    .setBootstrapServers("localhost:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("alerts")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("alerts-job")
    .build();

Flink also exposes Kafka to its SQL layer, so a topic can be declared as a table with CREATE TABLE ... WITH ('connector' = 'kafka', ...) and queried with SQL.

Flink stores the consumed offsets in its own checkpoints, together with operator state, and takes them periodically with a distributed snapshot algorithm. Committing offsets back to Kafka is optional and only for monitoring. Flink also has rich event-time support: watermarks tell it how late records may arrive before a window closes.

Exactly-once with a Kafka sink

Both engines can give end-to-end exactly-once into Kafka, using the transactions you already know:

  • Flink: DeliveryGuarantee.EXACTLY_ONCE makes the sink write inside a Kafka transaction that is committed when the matching checkpoint completes (a two-phase commit). Consumers of the output must use isolation.level=read_committed, or they see uncommitted records. Set the transaction timeout larger than the checkpoint interval plus restart time, and transaction.max.timeout.ms on the broker must allow it.
  • Spark: the Kafka sink is at-least-once: a replayed batch can write duplicates. Make writes idempotent downstream (a key plus dropDuplicates, or an idempotent target) to get effectively-once results.
JS Transactional sink: commit on checkpoint

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Which one?

Kafka Streams Spark Structured Streaming Flink
Runs as library in your app cluster job cluster job
Latency low micro-batch (continuous mode is limited) low, record at a time
Strong point simple Kafka-to-Kafka services batch + stream, ML, lakehouse event time, large state, low latency
Kafka sink guarantee exactly-once at-least-once exactly-once

If you only move data between topics, Streams is usually simplest. If your team already runs Spark, use it. For latency-sensitive, stateful, event-time-heavy jobs, Flink is the usual choice.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does a Spark Structured Streaming job resume after a crash?

    Spark records the offset range of each batch in the checkpoint and replays from there. It does not depend on consumer-group commits.
  2. 2. What does startingOffsets control in a Spark Kafka source?

    It matters only on the first run. After that the checkpoint decides.
  3. 3. What must consumers of a Flink exactly-once Kafka sink set?

    Records are written in a transaction that is committed when a checkpoint completes. read_committed hides records from open or aborted transactions.
  4. 4. Which statement about the Spark Kafka sink is correct?

    A replayed micro-batch can write the same records again, so make the result idempotent.

Summary

  • Spark and Flink are separate engines for jobs that outgrow a library or need many sources and sinks.
  • Spark reads Kafka as an unbounded table; progress lives in the checkpoint, not in a consumer group.
  • Flink's KafkaSource and KafkaSink keep offsets in checkpoints and support event time and exactly-once writes through transactions.
  • Exactly-once into Kafka needs read_committed readers; Spark's sink is at-least-once, so make the output idempotent.
  • Pick by ecosystem and needs: Streams for simple Kafka-to-Kafka, Spark for batch-plus-stream, Flink for event time and big state.

Next: review how clusters are run day to day in Cluster Architecture and KRaft.