Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 66 ยท Production Features

Kafka with Spring Boot in Practice

Create Kafka topics from Spring Boot, send and receive JSON, understand consumer groups and partitions, add retries with a dead-letter topic and start or stop listeners at run time.

Messaging and Events Between Services explained why a broker helps and which delivery guarantees you can expect. This lesson is the hands-on follow-up for Apache Kafka: how topics are created from your application, how JSON is serialized, how consumer groups share partitions, what happens when a message fails, and how to start and stop a listener while the application is running.

No Kafka broker runs on this site, so the Spring code is read-only. The behavior that is easiest to get wrong (partitions, groups, retries, dead-letter topics) is rebuilt as small runnable Java stand-ins that say what is real and what is simplified.

A Kafka pipeline in Spring Boot

Topic Declared as a bean, created on startup.
Producer KafkaTemplate with a key and JSON value.
Consumer group Members share the partitions.
Dead-letter topic Poison messages are parked, not looped.

Each piece has a Boot default you can override.

1 / 2

Creating topics from code

Spring for Apache Kafka includes a KafkaAdmin, and Boot auto-configures it. At startup it looks for NewTopic beans and creates any topic that does not exist yet:

Java
@Configuration
class KafkaTopics {

    @Bean
    NewTopic ordersPlaced() {
        return TopicBuilder.name("orders.placed")
                .partitions(3)
                .replicas(1)               // 1 for local development, usually 3 in production
                .config("retention.ms", "604800000")   // keep messages for 7 days
                .build();
    }
}

Three facts to remember. The partition count decides the maximum parallelism of one consumer group (more consumers than partitions leaves some idle), and you can raise it later but never lower it. A new partition count also changes which partition a given key hashes to, which breaks per-key ordering for existing keys. Replicas cannot exceed the number of brokers. And topic creation at startup is convenient for development; many teams create topics through infrastructure code in production and turn the beans off.

Producing and consuming JSON

Kafka stores bytes. Spring configures a serializer for keys and values on the producer and a deserializer on the consumer:

YAML
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      properties:
        enable.idempotence: true
    consumer:
      group-id: shipping
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.trusted.packages: com.example.shop.events

The class names depend on your version: Spring for Apache Kafka 3 uses JsonSerializer/JsonDeserializer, version 4 (with Jackson 3 in Boot 4) renames them to JacksonJsonSerializer/JacksonJsonDeserializer. Check the reference of the version you use.

Two settings deserve a comment. ErrorHandlingDeserializer wraps the real deserializer so a corrupt message becomes a recoverable error instead of an exception that blocks the partition forever. spring.json.trusted.packages limits which classes JSON may be turned into, a security measure against deserializing arbitrary types.

Java
@Service
class OrderPublisher {
    private final KafkaTemplate<String, OrderPlaced> kafka;

    OrderPublisher(KafkaTemplate<String, OrderPlaced> kafka) { this.kafka = kafka; }

    void publish(OrderPlaced event) {
        kafka.send("orders.placed", String.valueOf(event.customerId()), event)
             .whenComplete((result, error) -> {
                 if (error != null) { /* log and alert: the broker did not accept it */ }
             });
    }
}

@Component
class ShippingListener {
    @KafkaListener(id = "shipping", topics = "orders.placed", groupId = "shipping")
    void on(OrderPlaced event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        // create the shipment; this method may run twice for the same event
    }
}

The key (customerId) decides the partition, so all events of one customer stay in order. Because delivery is at-least-once, the handler must be idempotent, as explained in the previous lesson.

Consumer groups and partitions

A consumer group is a set of consumers that share one subscription. Kafka assigns each partition to exactly one member of a group, and every group tracks its own offsets. Two practical consequences follow:

  • Start a second instance of your service with the same group-id and the partitions are split between the two (a rebalance). That is how you scale reading.
  • A service with a different group-id receives every message again, independently. That is how several teams consume the same topic.

This stand-in models both rules. It hashes keys to partitions (Kafka really uses a murmur2 hash, so the exact numbers differ), assigns partitions to group members, and rebalances when a member joins.

Java Partitions, keys and consumer groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Look at the output: every ada message lands in the same partition and is read in order, shipping and billing each see all six messages, and once billing has two members its three partitions are split between them (two and one). With three partitions a fourth member would get nothing.

When a message fails: retries and the dead-letter topic

Suppose a listener throws an exception. If Kafka did nothing, the same record would fail again and again and block every message behind it in that partition. Spring's DefaultErrorHandler handles this: it retries the record according to a back-off policy and, when retries are exhausted, hands it to a recoverer. The standard recoverer, DeadLetterPublishingRecoverer, publishes the failed record to a dead-letter topic (by default <topic>.DLT, same partition number) together with headers describing the exception.

Java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);       // writes to orders.placed.DLT
    var backOff = new ExponentialBackOffWithMaxRetries(4);             // 4 retries: 1s, 2s, 4s, 8s
    backOff.setInitialInterval(1_000);
    backOff.setMultiplier(2.0);

    var handler = new DefaultErrorHandler(recoverer, backOff);
    handler.addNotRetryableExceptions(IllegalArgumentException.class); // bad data will never succeed
    return handler;
}

Boot picks up a single CommonErrorHandler bean and applies it to the listener containers. Blocking retries like this pause the partition while they wait, which is fine for short delays. For long delays without blocking, Spring offers non-blocking retry topics with @RetryableTopic, which move the record through -retry-0, -retry-1 topics before the DLT.

A dead-letter topic is only useful if somebody looks at it. Alert when it receives messages, keep enough information to understand the failure, and have a plan to fix and replay the records.

Starting and stopping a listener

Sometimes a listener must not start with the application (a maintenance window) or must be paused (the database is down, a downstream system is overloaded). Give the listener an id and use the KafkaListenerEndpointRegistry:

Java
@Component
class ShippingListener {
    @KafkaListener(id = "shipping", topics = "orders.placed", autoStartup = "false")
    void on(OrderPlaced event) { /* ... */ }
}

@RestController
@RequestMapping("/admin/listeners")
class ListenerAdmin {
    private final KafkaListenerEndpointRegistry registry;
    ListenerAdmin(KafkaListenerEndpointRegistry registry) { this.registry = registry; }

    @PostMapping("/{id}/start")
    void start(@PathVariable String id) { registry.getListenerContainer(id).start(); }

    @PostMapping("/{id}/pause")
    void pause(@PathVariable String id) { registry.getListenerContainer(id).pause(); }

    @PostMapping("/{id}/resume")
    void resume(@PathVariable String id) { registry.getListenerContainer(id).resume(); }

    @PostMapping("/{id}/stop")
    void stop(@PathVariable String id) { registry.getListenerContainer(id).stop(); }
}

stop() leaves the consumer group (which triggers a rebalance), while pause() keeps the membership and simply stops fetching. Protect admin endpoints like these with Spring Security; anyone who can call them can stop your processing.

The second stand-in combines everything: a container that retries a failing record, parks it in a dead-letter topic, and a registry that stops and restarts the listener while messages keep arriving. Offsets are kept, so nothing is lost while it is stopped.

Java Retries, a dead-letter topic and a listener registry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

What is real: the order of events (retry, then recover to the DLT, then commit) and the "resume from the committed offset" behavior. What is the stand-in: there is no back-off timing, no partitions or rebalancing, and the DLT is a list instead of a topic.

Practical checklist

  • Name topics and groups deliberately, for example orders.placed and a group per consuming service.
  • Choose keys for ordering, and remember that more partitions later reshuffle keys.
  • Use acks=all and idempotent producers for important events, and keep consumers idempotent.
  • Wrap deserializers with ErrorHandlingDeserializer, limit trusted packages and version your events.
  • Always configure a dead-letter path and watch it, together with consumer lag (Observability).
  • Test against a real broker with Testcontainers (Integration Tests with Testcontainers); mocks hide serialization and rebalancing behavior.

For a complete flow built on these ideas, see Project: Event-Driven Order Flow.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Three partitions, one consumer group with five instances. How many instances receive messages?

    Within a group each partition is read by exactly one member, so parallelism is capped by the partition count.
  2. 2. Why is the message key important?

    The key is hashed to choose a partition, and Kafka only guarantees order inside one partition.
  3. 3. What does DeadLetterPublishingRecoverer do?

    It is the recovery step of DefaultErrorHandler, so a poison message no longer blocks the partition and can be inspected later.
  4. 4. How do you prevent a listener from starting automatically and start it later?

    The registry holds every listener container by id, so you can start, stop, pause and resume it at run time.
  5. 5. A second service uses a different group-id on the same topic. What happens?

    Groups are independent subscriptions. Only members of the same group split the partitions.

Summary

  • Declare NewTopic beans (via TopicBuilder) and Boot's KafkaAdmin creates missing topics; partitions can grow but never shrink.
  • Serialize keys as strings and values as JSON, wrap the deserializer in ErrorHandlingDeserializer and restrict trusted packages.
  • Partitions give order per key; a consumer group splits partitions among its members, and different groups read independently.
  • DefaultErrorHandler with a back-off and DeadLetterPublishingRecoverer retries failing records and parks the rest in a dead-letter topic; @RetryableTopic gives non-blocking retries.
  • KafkaListenerEndpointRegistry starts, stops, pauses and resumes listeners by id.

Next: Microservices with Spring Cloud.