Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 39 ยท Operations

Testing Kafka Applications

Test Kafka producers, consumers and Streams topologies: unit tests with mocks, TopologyTestDriver, Testcontainers with a real broker, and contract tests, with runnable stand-ins.

Code that talks to Kafka is hard to test in the usual way: it depends on a running cluster, on timing, on partitions and on offsets. Teams often respond by testing nothing below the full environment, and then bugs in serialization, partitioning and error handling reach production.

A better plan is a test pyramid: many fast unit tests that never touch a broker, a smaller set of integration tests against a real broker in a container, and a few contract checks for the data formats. This lesson shows each layer.

Real versus stand-in. The Java snippets are real usage of Kafka's test utilities, read-only on this site. The TryIt blocks are plain JavaScript stand-ins that follow the same testing ideas (a mock producer, a topology driven without a broker, a compatibility check). They are not the Kafka libraries.

Separate logic from I/O

The cheapest improvement costs nothing: keep business rules in plain functions and classes, and keep the Kafka calls thin. A function Order enrich(Order o) can be tested with ordinary unit tests. Only the small piece that calls producer.send or consumer.poll needs Kafka-specific testing.

Unit tests with MockProducer and MockConsumer

The kafka-clients library ships MockProducer and MockConsumer. They implement the same interfaces as the real ones but keep records in memory.

Java
MockProducer<String, String> producer =
    new MockProducer<>(true, new StringSerializer(), new StringSerializer());

OrderPublisher publisher = new OrderPublisher(producer, "orders");
publisher.publish(new Order("customer-42", 1001));

List<ProducerRecord<String, String>> sent = producer.history();
assertEquals(1, sent.size());
assertEquals("customer-42", sent.get(0).key());

The first argument, autoComplete, makes every send succeed at once. With false you call producer.completeNext() or producer.errorNext(new RuntimeException("boom")) yourself, which is how you test the failure path of your callback.

MockConsumer works in the opposite direction: you assign partitions, set beginning offsets and add records, then call your processing code.

Java
MockConsumer<String, String> consumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
TopicPartition tp = new TopicPartition("orders", 0);
consumer.assign(List.of(tp));
consumer.updateBeginningOffsets(Map.of(tp, 0L));
consumer.addRecord(new ConsumerRecord<>("orders", 0, 0L, "customer-42", "{...}"));

processor.pollOnce(consumer);
assertEquals(1, processor.handledCount());

Design your classes so the producer or consumer is passed in (constructor injection). That is what lets a test substitute the mock.

JS A mock producer for unit tests

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Testing Kafka Streams with TopologyTestDriver

A Kafka Streams topology can be tested without a broker. TopologyTestDriver runs your topology synchronously in the test thread. You pipe input records in and read output records out.

Java
Topology topology = builder.build();
try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) {
    TestInputTopic<String, String> in =
        driver.createInputTopic("lines", new StringSerializer(), new StringSerializer());
    TestOutputTopic<String, Long> out =
        driver.createOutputTopic("counts", new StringDeserializer(), new LongDeserializer());

    in.pipeInput("hello kafka hello");
    assertEquals(Map.of("hello", 2L, "kafka", 1L), out.readKeyValuesToMap());
}

It is fast and deterministic. Time does not pass by itself: you give records timestamps (or advance wall-clock time with driver.advanceWallClockTime) to test windows and punctuators. It does not model partitions, rebalances or real concurrency, so it cannot find bugs in those areas.

JS Driving a topology without a broker

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Integration tests with a real broker

Mocks cannot tell you whether your configuration works: serializers registered correctly, acks, consumer group behaviour, topic auto-creation, transactions. For that you need a real broker. The practical choice is Testcontainers, which starts a Kafka container for the test and removes it afterwards.

Java
@Testcontainers
class OrderFlowIT {

    @Container
    static KafkaContainer kafka =
        new KafkaContainer(DockerImageName.parse("apache/kafka:3.9.0"));

    @Test
    void ordersFlowThroughTheTopic() {
        String bootstrap = kafka.getBootstrapServers();
        // create the topic with an AdminClient, produce with the real producer,
        // then poll with a real consumer and assert on the records
    }
}

With Spring Boot, @EmbeddedKafka starts an in-JVM broker (KRaft mode in current versions), which avoids needing Docker but is a little further from production. A few tips make these tests reliable:

  • Never sleep. Poll with a deadline, for example with the Awaitility library, until the expected record arrives or the timeout passes.
  • Use a fresh topic name per test (add a random suffix) and a fresh group.id, so tests do not see each other's data.
  • Set auto.offset.reset=earliest so a new consumer sees records produced before it joined.
  • Share one container across the test class or suite; starting a broker is the slow part.
  • Test the unhappy paths: a record that fails deserialization, a consumer that throws, a broker restart.

Contract tests for data formats

The biggest source of production breakage between teams is a schema change. A producer renames a field and every consumer crashes. If you use a schema registry (see Schemas, Avro, Protobuf and JSON), make compatibility a build step:

  • Keep schemas in version control, and in CI ask the registry whether the new schema is compatible with the registered ones before merging (the registry has a compatibility-check endpoint).
  • Keep sample messages as test fixtures. A consumer test deserializes every fixture; a producer test checks that its output validates against the schema.
  • Use consumer-driven contract tools when many teams share a topic.
JS A backward-compatibility check

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Choosing the layer

Layer Speed Finds
Plain unit tests of business logic milliseconds logic bugs
MockProducer / MockConsumer milliseconds send/poll handling, error paths
TopologyTestDriver milliseconds Streams logic, windows, joins
Testcontainers / embedded broker seconds configuration, serialization, real behaviour
Schema compatibility checks seconds breaking format changes

Keep the base wide and the top narrow. A handful of integration tests that cover each topic's main flow is usually enough.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of MockProducer over a real producer in a unit test?

    MockProducer needs no broker, exposes the records sent, and with autoComplete off lets you complete or fail sends yourself.
  2. 2. What does TopologyTestDriver NOT test well?

    The driver runs synchronously without a real cluster, so rebalances and concurrency need a real-broker test.
  3. 3. How should an integration test wait for a record to arrive?

    Fixed sleeps are slow and flaky. Polling with a timeout is fast when things work and still bounded when they do not.
  4. 4. Why use a random topic name and group.id per integration test?

    Shared topics and groups make tests depend on execution order. Fresh names isolate them.

Summary

  • Keep business logic separate from Kafka calls and inject the producer or consumer so mocks can replace them.
  • MockProducer and MockConsumer give fast unit tests including failure injection.
  • TopologyTestDriver tests Kafka Streams topologies without a broker, with controlled time.
  • Use Testcontainers or an embedded broker for a few integration tests; poll instead of sleeping, and isolate topics and groups per test.
  • Run schema compatibility checks and fixture-based contract tests in CI.

Next: see how change flows end to end in Kafka Connect, another piece you can test in a containerised setup.