Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 39 ยท Patterns

Kafka with Spring Boot

Use Apache Kafka from Spring Boot: KafkaTemplate, @KafkaListener, application.yml settings, NewTopic beans, retries and dead-letter topics, with a runnable stand-in.

Most Java services that talk to Kafka are Spring Boot applications. Spring for Apache Kafka (the spring-kafka library) wraps the plain producer and consumer clients you met in Writing a Producer and Writing a Consumer. You write a method, annotate it, and the framework runs the poll loop, deserializes records, commits offsets and retries failures for you.

This lesson shows the pieces you use every day: configuration, KafkaTemplate for sending, @KafkaListener for receiving, topic creation with beans, and error handling with a dead-letter topic. Then it ends with a small REST-to-Kafka service.

Real versus stand-in. The Spring and YAML code below is real usage, shown read-only because the site cannot run Spring or reach a Kafka cluster. The TryIt blocks are plain JavaScript stand-ins that copy the shape of KafkaTemplate and a listener container (a poll loop that calls your method and retries). They are not Spring.

Dependencies and configuration

Add the starter and the Kafka support comes with auto-configuration:

XML
<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

Almost everything is configured under spring.kafka in application.yml:

YAML
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      acks: all
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        enable.idempotence: true
    consumer:
      group-id: order-service
      auto-offset-reset: earliest
      enable-auto-commit: false
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: com.example.orders
    listener:
      ack-mode: record
      concurrency: 3

Anything the plain clients accept (see batching and idempotence and offsets and commits) can go under properties:. The settings above mean: wait for all in-sync replicas, avoid duplicates on retry, start from the beginning when the group has no committed offset, and let Spring commit after each record is handled.

Creating topics with beans

Spring can create topics at startup if you declare them. A KafkaAdmin bean is auto-configured, and every NewTopic bean becomes a createTopics call (see The AdminClient API):

Java
@Configuration
public class KafkaTopics {

    @Bean
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
            .partitions(6)
            .replicas(3)
            .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7 days
            .build();
    }
}

If the topic already exists with the same name, nothing happens. Spring will add partitions if you raise the number, but it never shrinks a topic. Many teams let infrastructure code own topics in production and keep these beans for local development and tests.

Sending with KafkaTemplate

KafkaTemplate<K, V> is Spring's producer. It is thread-safe, so inject one and share it.

Java
@Service
public class OrderPublisher {

    private final KafkaTemplate<String, OrderPlaced> template;

    public OrderPublisher(KafkaTemplate<String, OrderPlaced> template) {
        this.template = template;
    }

    public CompletableFuture<SendResult<String, OrderPlaced>> publish(OrderPlaced event) {
        return template.send("orders", event.customerId(), event)
            .whenComplete((result, error) -> {
                if (error != null) {
                    log.error("send failed for order {}", event.id(), error);
                } else {
                    log.info("stored at {}", result.getRecordMetadata());
                }
            });
    }
}

send(topic, key, value) returns a CompletableFuture that completes when the broker answers, which is the callback style from the producer lesson. Using the customer id as the key keeps one customer's events in one partition and therefore in order (see Message Keys and Partitioners).

Receiving with @KafkaListener

Java
@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(OrderPlaced event,
                        @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
                        @Header(KafkaHeaders.OFFSET) long offset) {
        log.info("order {} from partition {} offset {}", event.id(), partition, offset);
        // business logic here
    }
}

Spring starts a listener container for this method. It owns a consumer, polls in a loop, converts each record with the configured deserializer and calls your method. If the method returns normally, the offset is marked for commit according to ack-mode. concurrency: 3 starts three consumer threads in the same group, so the group has three members and the partitions are shared among them. More threads than partitions never helps (see Consumer Groups).

You can also take the whole ConsumerRecord<String, OrderPlaced>, a List of values for batch listeners, or an Acknowledgment argument when you set ack-mode: manual and want to commit yourself.

Stand-in: a template and a listener container

The stand-in below keeps a topic in memory. template.send appends a record and returns a promise of the partition and offset. The container object polls from the group's position, calls the listener and then commits, which is what ack-mode: record does.

JS KafkaTemplate and a listener container

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Errors, retries and the dead-letter topic

By default, when a listener throws, Spring retries the record a few times and then logs and skips it, which is risky. Configure a DefaultErrorHandler with a back-off and a recoverer that publishes the failed record to a dead-letter topic (DLT), the pattern from Retries and Dead-Letter Topics:

Java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template); // sends to <topic>.DLT
    var backOff = new FixedBackOff(1000L, 3);                    // 3 retries, 1 s apart
    var handler = new DefaultErrorHandler(recoverer, backOff);
    handler.addNotRetryableExceptions(IllegalArgumentException.class);
    return handler;
}

Spring also offers @RetryableTopic, which creates retry topics with growing delays plus a DLT, and a @DltHandler method to process the dead letters. Exceptions that can never succeed, like a malformed payload, should be marked not retryable so they go to the DLT at once.

JS Retry then dead-letter

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A REST-to-Kafka service

Putting the pieces together, a controller accepts an HTTP request and publishes an event; a listener elsewhere handles it:

Java
@RestController
@RequestMapping("/orders")
class OrderController {

    private final OrderPublisher publisher;

    OrderController(OrderPublisher publisher) { this.publisher = publisher; }

    @PostMapping
    ResponseEntity<Void> place(@RequestBody OrderPlaced order) {
        publisher.publish(order);
        return ResponseEntity.accepted().build();   // 202: queued, not processed yet
    }
}

Return 202 Accepted, not 201 Created: the order is only queued. If clients must know the publish succeeded, wait for the future (publisher.publish(order).get(5, TimeUnit.SECONDS)) before answering. For stronger guarantees across a database and Kafka together, see the outbox idea in Event-Driven Patterns.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a @KafkaListener method rely on Spring to run?

    Spring starts a listener container that owns the consumer and its poll loop, then calls your method for each record.
  2. 2. What does listener concurrency: 3 do?

    Each thread is a group member, so the topic's partitions are shared among three consumers. It cannot exceed the partition count usefully.
  3. 3. A record can never be processed because its payload is malformed. What is the best handling?

    Retrying cannot fix bad data. Sending it to a DLT at once keeps the partition moving and keeps the record for inspection.
  4. 4. Why return 202 Accepted from a REST endpoint that only publishes an event?

    202 says the request was accepted for processing. The real result happens later in a consumer.

Summary

  • spring-kafka auto-configures producers, consumers and a KafkaAdmin from spring.kafka.* properties.
  • KafkaTemplate.send is the asynchronous producer; the key decides the partition.
  • @KafkaListener runs a listener container; concurrency adds consumer threads and ack-mode controls commits.
  • NewTopic beans create topics at startup, usually only for development.
  • Use DefaultErrorHandler with a back-off and DeadLetterPublishingRecoverer, or @RetryableTopic, instead of the default log-and-skip.

Next: calling Kafka from another language in Kafka from Python.