Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 39 ยท Data Formats and the Ecosystem

Kafka Streams and Stream Processing

Learn Kafka Streams and stream processing: KStream vs KTable, stateless and stateful operations, windowing, joins and state stores with changelog topics, with runnable stand-ins.

A consumer plus a producer can transform data: read a topic, change each record, write another topic. But the moment you want to count events per user, join two topics or work in time windows, you need state, and state brings hard problems: what happens when the process crashes, when you add instances, or when events arrive late? Stream processing frameworks solve those problems once so you do not solve them by hand in every service.

Kafka Streams is the stream processing library that ships with Apache Kafka. It is a plain Java library, not a cluster: your application embeds it, and you scale by starting more copies of the same application. This lesson explains its concepts and shows runnable stand-ins for the ideas. It builds on Kafka Connect, Log Compaction and Tombstones and Consumer Groups and Rebalancing.

Real versus stand-in. The Java code below is real Kafka Streams usage, shown read-only because the site cannot run Java against a Kafka cluster. The TryIt blocks are plain JavaScript stand-ins that model the semantics (streams versus tables, windows, joins, state restore) with arrays and maps. They are not the library.

Two views of the same data

KStream Every record is an event.
KTable Latest value per key.

A table is a stream that was folded by key.

1 / 3

A small Streams application

A Streams program describes a topology: a graph of processing steps between input and output topics. This one counts page views per page in one-minute windows:

Java
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "page-view-counter");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);

StreamsBuilder builder = new StreamsBuilder();

KStream<String, String> views = builder.stream("page-views");   // key = user, value = page

KTable<Windowed<String>, Long> counts = views
    .filter((user, page) -> page != null && !page.isEmpty())    // stateless
    .groupBy((user, page) -> page)                              // re-key by page
    .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(1), Duration.ofSeconds(10)))
    .count();                                                   // stateful

counts.toStream()
      .map((window, count) -> KeyValue.pair(window.key() + "@" + window.window().start(), count))
      .to("page-view-counts", Produced.with(Serdes.String(), Serdes.Long()));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

application.id doubles as the consumer group id and as a prefix for internal topics. Running two copies of this program with the same id splits the input partitions between them, exactly like a consumer group. Serdes (serializer plus deserializer pairs) tell Streams how to turn bytes into objects and back (see Schemas, Avro, Protobuf and JSON).

KStream versus KTable

A KStream is an unbounded sequence of independent events. Two records with the same key are two events: both count. A KTable is a changelog interpreted as a table: a later record with the same key replaces the earlier one, and a null value deletes the key. It is the compaction idea from the previous chapter, applied inside a processor. Summing the same four records shows the difference.

JS The same records as a stream and as a table

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Use a stream for things that happened (clicks, payments) and a table for things that currently are (a profile, a price). Aggregations turn a stream into a table: count() on a grouped stream yields a KTable<key, count> that updates with every event.

Stateless and stateful operations

Stateless operations look at one record at a time and need no memory: filter, map, mapValues, flatMap, selectKey, peek, branch-style splitting. They are cheap and trivially scalable.

Stateful operations need history: count, reduce, aggregate, windowing and joins. Their state lives in a state store, a local key-value database (RocksDB by default) next to the processing thread, so lookups are fast and need no network call.

One subtlety: groupBy with a new key means records must move to the partition that owns that key, so Streams writes them to an internal repartition topic and reads them back. This costs extra network and disk, so prefer groupByKey when the key is already right.

Windowing

Counting "events so far" never ends; usually you want "events in each minute". Windows cut the stream into time buckets, using the event time (the timestamp in the record) rather than wall-clock time:

  • Tumbling: fixed size, no overlap ([0,60), [60,120)).
  • Hopping: fixed size, overlapping (size 5 minutes, advance 1 minute).
  • Sliding: a window per pair of records that fall within the size.
  • Session: grouped by activity, closing after an inactivity gap.

Events can arrive late. A grace period says how long after a window's end Streams still accepts late records for it; after that, the window is closed and later arrivals are dropped. The clock for this is stream time, the largest timestamp seen so far, not the wall clock.

JS Tumbling windows with a grace period

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The event at t=58 arrived after t=65 but still counts, because stream time (65) had not passed 70. The event at t=40 came when stream time was 80, so its window was already closed. Choosing the grace period is a trade: longer means fewer dropped events and later results.

Joins

Streams supports three families of joins:

  • Stream-stream joins match events from two streams that happen within a time window of each other (a click and the purchase that followed within 30 minutes).
  • Stream-table joins enrich each event with the current row of a table (an order with the customer's profile). The stream side triggers output; table updates do not.
  • Table-table joins keep two tables joined; the result is a table that updates whenever either side changes.

Joins need the inputs to be co-partitioned: the same number of partitions and the same partitioning of keys, so matching keys sit in the same task. If they are not, re-key and repartition first. A GlobalKTable is the exception: every instance holds a full copy, so any key can be joined without co-partitioning, at the cost of memory.

Java
KStream<String, String> clicks = builder.stream("clicks");
KTable<String, String> users = builder.table("users");     // key = user id, value = city
clicks.join(users, (page, city) -> page + " from " + city)
      .to("enriched-clicks");

State stores and changelog topics

What if the machine holding a state store dies? Every state store is backed by a changelog topic in Kafka: each update to the store is also written there. The changelog topic is compacted (see Log Compaction and Tombstones), so it holds the latest value per key. A replacement instance replays the changelog into a fresh local store before it resumes processing. To shorten that restore, num.standby.replicas keeps warm copies of stores on other instances.

JS Join and restore from a changelog

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Streams can also provide exactly-once results end to end inside Kafka: set processing.guarantee=exactly_once_v2 and it uses the transactions from Transactions and Exactly-Once for you.

Hands-on: word count, topologies and testing

Word count is the "hello world" of stream processing. Lines of text arrive on one topic; counts per word leave on another.

Java
StreamsBuilder builder = new StreamsBuilder();

builder.<String, String>stream("text-lines")
    .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+")))
    .filter((key, word) -> !word.isEmpty())
    .groupBy((key, word) -> word)          // re-key by word: triggers a repartition topic
    .count(Materialized.as("word-counts")) // KTable<String, Long> backed by a named store
    .toStream()
    .to("word-counts-out", Produced.with(Serdes.String(), Serdes.Long()));

Topology topology = builder.build();
System.out.println(topology.describe());   // prints sub-topologies, processors and stores

groupBy changes the key, so records with the same word must be brought to the same partition. Streams does that with an internal repartition topic, which is why describe() shows two sub-topologies: one reads the input, the other reads the repartition topic and owns the word-counts store. Printing the description is the quickest way to see where the state and the extra topics are before you deploy.

JS Word count with a repartition step

Press Run (or Ctrl + Enter) to see the console output here.

No output.

GlobalKTable

A normal KTable is partitioned like its topic, so joining a stream to it needs both sides to be co-partitioned by the same key. A GlobalKTable is fully copied to every application instance. It suits small reference data (countries, product names), lets you join on any field through a key-mapper function, and needs no co-partitioning. The cost is memory and disk on every instance, so keep it small.

Java
GlobalKTable<String, String> products = builder.globalTable("products");

orders.join(products,
    (orderKey, order) -> order.productId(),     // pick the lookup key from the stream record
    (order, productName) -> order.withName(productName));

The Processor API

The DSL covers most work. The lower-level Processor API gives you a processor with direct access to state stores and to time, so you can write logic the DSL has no operator for. This one emits an alert when a key's count passes a threshold, and clears it every minute:

Java
public class ThresholdProcessor implements Processor<String, Long, String, String> {
    private ProcessorContext<String, String> context;
    private KeyValueStore<String, Long> store;

    @Override
    public void init(ProcessorContext<String, String> context) {
        this.context = context;
        this.store = context.getStateStore("seen");
        context.schedule(Duration.ofMinutes(1), PunctuationType.WALL_CLOCK_TIME,
            ts -> store.all().forEachRemaining(kv -> store.delete(kv.key)));
    }

    @Override
    public void process(Record<String, Long> record) {
        long total = store.get(record.key()) == null ? record.value() : store.get(record.key()) + record.value();
        store.put(record.key(), total);
        if (total > 100) {
            context.forward(record.withValue("over threshold: " + total));
        }
    }
}
// topology.addProcessor("threshold", ThresholdProcessor::new, "source")
//         .addStateStore(storeBuilder, "threshold");

You can also mix the two: stream.process(...) runs a processor inside a DSL topology.

Testing without a broker: TopologyTestDriver

TopologyTestDriver (in kafka-streams-test-utils) runs a topology in memory, in one thread, with no broker. You pipe records in and read what comes out, which makes unit tests fast and deterministic.

Java
try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) {
    TestInputTopic<String, String> in =
        driver.createInputTopic("text-lines", new StringSerializer(), new StringSerializer());
    TestOutputTopic<String, Long> out =
        driver.createOutputTopic("word-counts-out", new StringDeserializer(), new LongDeserializer());

    in.pipeInput("to be or not to be");
    assertEquals(Map.of("to", 2L, "be", 2L, "or", 1L, "not", 1L), out.readKeyValuesToMap());
}

Windows and punctuators depend on time, so the driver lets you set the record timestamp (pipeInput(key, value, instant)) and advance wall-clock time (advanceWallClockTime) instead of sleeping. The stand-in above is the same idea: a pure function from input to output is easy to test. More on testing in Testing Kafka Applications.

Other tools

Kafka Streams is not the only choice. ksqlDB lets you write the same kinds of pipelines in SQL and runs as its own server (see ksqlDB: SQL over Kafka). Apache Flink (and Spark Structured Streaming) are separate cluster engines with richer features for large, multi-source jobs. Kafka Streams is a good fit when you want a library inside a JVM service with Kafka as both the source and the sink; reach for the others when the surrounding ecosystem demands it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Four records arrive with keys alice, bob, alice, bob. What is the difference between reading them as a KStream and as a KTable?

    In a stream, two records with the same key are two events. In a table the later record replaces the earlier one.
  2. 2. What is a grace period in windowed aggregation?

    After a window's end plus the grace period (measured in stream time) the window is closed and later arrivals are dropped.
  3. 3. Why does every state store have a changelog topic?

    The compacted changelog holds the latest value per key, so a new instance can replay it and restore the local store.
  4. 4. What must be true for a stream-table join to work?

    Matching keys must be handled by the same task, which requires co-partitioned inputs. A GlobalKTable avoids that requirement but copies all data to each instance.

Summary

  • Kafka Streams is a Java library for stateful stream processing that scales by running more instances; the topology is built from Kafka topics in and out.
  • A KStream is a sequence of events, a KTable is the latest value per key, and aggregations turn streams into tables.
  • Windows group events by event time; a grace period decides how late events can be.
  • Joins need co-partitioned inputs (or a GlobalKTable); groupBy with a new key causes a repartition topic.
  • State stores are local and fast, backed by compacted changelog topics for recovery; exactly_once_v2 adds end-to-end guarantees inside Kafka.
  • ksqlDB and Flink are alternatives for SQL-style or larger-scale processing.

Next, learn how to run the brokers behind all of this in Running a Cluster: KRaft, Brokers and Controllers.