Skip to content
Tutorials Center
My learning

Topics

Lesson 93 of 119 ยท Architecture and Deployment

Messaging with Kafka and RabbitMQ

Learn asynchronous messaging in Java - queues versus topics, Kafka partitions and consumer groups, RabbitMQ exchanges, delivery guarantees, idempotency and the outbox.

When the loan service records a new loan, other parts of the system care: the reminder service schedules a due-date email, the statistics service updates a dashboard, the recommendations service learns a new preference. The loan service could call all three over HTTP. Then it would wait for each one, fail when any of them is down, and need a code change every time a new interested service appears.

Messaging removes that coupling. The loan service publishes one message, LoanCreated, to a message broker and moves on. Each interested service receives the message when it is ready, even if it was down for maintenance when the loan happened. The broker stores messages durably in between. The two brokers you meet most in Java systems are Apache Kafka and RabbitMQ.

In this lesson you will learn the two core models (queues and topics), how Kafka and RabbitMQ implement them, what "at-least-once delivery" really means for your code, and how to publish messages without losing them. The brokers can't run here, so you will build their key behaviors in plain Java with BlockingQueue, threads and H2, and then see the Spring Kafka and Spring AMQP code you would write in production. It helps to have read Microservices.

Why asynchronous messaging?

  • Temporal decoupling. Producer and consumer don't need to be running at the same time. If the reminder service is down for an hour, messages wait for it.
  • Load leveling. A burst of 10,000 orders at noon fills the queue; consumers work through it at their own pace instead of collapsing.
  • Fan-out without coupling. New consumers subscribe without the producer knowing about them.
  • Scalability. Add consumer instances to process more messages in parallel.

The price: the producer doesn't get an immediate answer, the system becomes eventually consistent, and debugging requires tracing messages across services. Use HTTP when the caller needs the answer now ("is this book available?"), and messaging when it only needs to announce that something happened ("a loan was created") or to hand off work ("generate this report").

Queues and topics

There are two fundamental delivery models:

  • Point-to-point (queue). Each message is processed by exactly one of the consumers listening to the queue. Several consumers on one queue are competing consumers: they share the work. Use it for jobs: "send this email", "resize this image".
  • Publish/subscribe (topic). Each message is delivered to every subscriber. Use it for events: "loan created" goes to reminders, statistics and recommendations.

The JDK's BlockingQueue is a perfect in-process model of a queue. put blocks when a bounded queue is full (natural backpressure for producers), and take blocks until a message is available. Three consumers compete for 12 jobs:

Java Competing consumers on a BlockingQueue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The split between consumers changes from run to run, which is the point: whoever is free takes the next job. Each job is still handled exactly once. Notice the poison pill, a special message that tells a consumer to stop; real brokers don't need it, but in-process queues often do.

An in-memory queue disappears when the JVM stops, and it can't cross machines. A broker gives you the same model durably and over the network.

RabbitMQ: exchanges, queues and bindings

RabbitMQ is a traditional message broker built around the AMQP protocol. Producers never write to queues directly; they publish to an exchange with a routing key, and bindings decide which queues receive a copy:

Exchange type Routing rule Typical use
direct Queue binding key equals the routing key Jobs by type: email, sms
topic Pattern match: * = one word, # = zero or more loan.created, loan.*, loan.#
fanout Every bound queue gets every message Broadcasts
headers Match on message headers Rarely used

So the same broker does both models: one queue with several consumers is point-to-point; one exchange bound to many queues (one per subscribing service) is publish/subscribe.

Consumers acknowledge each message after processing it. Until the ack arrives, RabbitMQ keeps the message; if the consumer dies, the message is redelivered to another consumer. Prefetch (basic.qos) limits how many unacknowledged messages a consumer holds at once. Messages that fail repeatedly can be routed to a dead-letter exchange. Once acknowledged, a message is gone from the queue: RabbitMQ is optimized for handing work out, not for keeping history. (RabbitMQ streams add a Kafka-like replayable log for cases that need it.)

With Spring AMQP (spring-boot-starter-amqp), declaring and using these takes a few beans:

Java
@Configuration
class RabbitConfig {
    @Bean TopicExchange libraryEvents() { return new TopicExchange("library.events"); }
    @Bean Queue reminderQueue() {
        return QueueBuilder.durable("reminders.loan-created").deadLetterExchange("library.dlx").build();
    }
    @Bean Binding reminderBinding(Queue reminderQueue, TopicExchange libraryEvents) {
        return BindingBuilder.bind(reminderQueue).to(libraryEvents).with("loan.created");
    }
    @Bean MessageConverter json() { return new Jackson2JsonMessageConverter(); }   // JSON message bodies
}

@Service
class LoanPublisher {
    private final RabbitTemplate rabbit;
    LoanPublisher(RabbitTemplate rabbit) { this.rabbit = rabbit; }

    void loanCreated(LoanCreated event) {
        rabbit.convertAndSend("library.events", "loan.created", event);
    }
}

@Component
class ReminderListener {
    @RabbitListener(queues = "reminders.loan-created")
    void on(LoanCreated event) {
        reminders.scheduleFor(event.loanId(), event.dueDate());
    }   // returning normally = ack; throwing = retry or dead-letter, depending on configuration
}

Kafka: a distributed log

Apache Kafka works differently. A Kafka topic is an append-only log, split into partitions. Producers append records to the end; each record gets an increasing offset within its partition. Consumers don't remove records; they remember how far they have read (their committed offset). Records stay for a configured retention period (days, or forever), so a new service can replay history from the beginning.

Three rules explain almost everything about Kafka:

  1. The record key picks the partition (by hashing the key). All records with the same key, such as the same bookId, land in the same partition.
  2. Order is guaranteed only within a partition. So all events for one book are processed in order, but events for different books may interleave.
  3. Consumers form groups. Within a consumer group, each partition is assigned to exactly one consumer, so the group shares the work (point-to-point). Different groups each read the whole topic independently (publish/subscribe). The number of partitions caps the parallelism of one group: with 6 partitions, a 7th consumer sits idle.

The next example is a miniature Kafka: a topic with partitions as lists, key-based partitioning, per-group offsets, and two consumer groups that read the same records independently. It also replays the log for a brand-new group.

Java A mini Kafka: partitions, offsets and consumer groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Look at book-1: its three events are all in the same partition with increasing offsets, so every group sees BORROWED, RETURNED, BORROWED in that order. The two groups processed the same records independently, the second poll only delivered the new record (thanks to committed offsets), and the new audit group read everything from offset 0. That replay ability is Kafka's signature feature: it is both a message broker and a durable event store.

Spring for Apache Kafka

With spring-kafka (in Boot 4, add spring-boot-starter-kafka, which brings it together with Boot's Kafka auto-configuration), Spring Boot configures producers and consumers from properties:

YAML
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      acks: all                       # wait until all in-sync replicas have the record
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: reminders
      auto-offset-reset: earliest     # a new group starts at the beginning of the log
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer

Add a JSON serializer and deserializer for values (Spring Kafka provides them; class names differ slightly between Spring Kafka 3 and the Jackson 3 based Spring Kafka 4). Then:

Java
public record LoanCreated(String eventId, long loanId, long bookId, long memberId, LocalDate dueDate, Instant occurredAt) {}

@Service
class LoanEventPublisher {
    private final KafkaTemplate<String, LoanCreated> kafka;
    LoanEventPublisher(KafkaTemplate<String, LoanCreated> kafka) { this.kafka = kafka; }

    void publish(LoanCreated event) {
        // The key decides the partition: all events of one book stay in order.
        kafka.send("library.loans", String.valueOf(event.bookId()), event);
    }
}

@Component
class ReminderConsumer {
    @KafkaListener(topics = "library.loans", groupId = "reminders", concurrency = "3")
    void on(LoanCreated event) {
        reminders.scheduleFor(event.loanId(), event.dueDate());
    }
}

concurrency = "3" starts three consumers in the group, useful when the topic has at least three partitions. By default, Spring commits offsets after your listener returns. If it throws, the DefaultErrorHandler retries with a back-off and can hand the record to a DeadLetterPublishingRecoverer, which writes it to a library.loans.DLT topic; @RetryableTopic sets up non-blocking retries through extra retry topics.

Kafka or RabbitMQ?

Kafka RabbitMQ
Model Partitioned, replayable log Broker with exchanges and queues
After consumption Records stay until retention expires Acknowledged messages are removed
Ordering Per partition (per key) Per queue, with one consumer
Routing By topic and key Flexible: direct, topic patterns, fanout, headers
Throughput Very high (millions of records/s per cluster) High
Best for Event streaming, event sourcing, analytics pipelines, many independent consumers Task queues, complex routing, request/reply, per-message features (priority, TTL)

Many companies use both. Cloud services such as Amazon SQS/SNS, Google Pub/Sub and Azure Service Bus offer the same models as managed services.

Kafka listener lifecycle and a local setup

A @KafkaListener method looks like a plain method, but Spring wraps it in a message listener container that owns a consumer thread, polls the broker, calls your method, commits offsets and handles errors. Knowing that container, and how to start, pause and stop it, answers practical questions: how do I stop consuming during a maintenance window, how do I avoid reading messages before my cache is warm, and what happens to a message that keeps failing?

The container and the registry

Every @KafkaListener is registered in a bean called the KafkaListenerEndpointRegistry. Give the listener an id and you can find its container, and with autoStartup = "false" it does not poll until you say so:

Java
@Component
class ReminderConsumer {

    @KafkaListener(id = "reminders", topics = "library.loans", groupId = "reminders", autoStartup = "false")
    void on(LoanCreated event) {
        reminders.scheduleFor(event.loanId(), event.dueDate());
    }
}

@Component
class ConsumerControl {

    private final KafkaListenerEndpointRegistry registry;

    ConsumerControl(KafkaListenerEndpointRegistry registry) { this.registry = registry; }

    public void start()  { registry.getListenerContainer("reminders").start(); }
    public void pause()  { registry.getListenerContainer("reminders").pause(); }    // stops fetching, stays in the group
    public void resume() { registry.getListenerContainer("reminders").resume(); }
    public void stop()   { registry.getListenerContainer("reminders").stop(); }     // leaves the group, triggers a rebalance
}

The difference between pause and stop is important:

pause() stop()
Consumer keeps polling the broker yes, but it fetches no records no, the consumer is closed
Stays a member of the group yes, keeps its partitions no, its partitions move to other consumers (a rebalance)
Resume cost immediate reconnect and rebalance
Use for short pauses, back-pressure, waiting for a dependency long maintenance, shutdown

A paused consumer must keep calling poll() (Spring does this for you), or the broker decides it is dead after max.poll.interval.ms and takes its partitions away. Spring Boot also stops the containers gracefully on shutdown, finishing the record in progress before it commits.

Offsets and acknowledgment

By default the container commits offsets after your method returns (ack-mode: batch). If you must control the commit yourself, for example to commit only after a database write succeeded, use manual acknowledgment:

YAML
spring:
  kafka:
    listener:
      ack-mode: manual
      concurrency: 3
Java
@KafkaListener(id = "reminders", topics = "library.loans")
void on(LoanCreated event, Acknowledgment ack) {
    reminders.scheduleFor(event.loanId(), event.dueDate());
    ack.acknowledge();                  // commit this offset now; if we crash before this line, the record is redelivered
}

Because a crash before the acknowledgment redelivers the record, the listener must be idempotent, as explained in the delivery guarantees section below.

Error handling and the dead-letter topic

When the listener throws, Spring's DefaultErrorHandler takes over. By default it retries the record a few times and then logs and skips it. In production you configure the back-off, mark exceptions that are not worth retrying, and send the failed record to a dead-letter topic (DLT) for later inspection:

Java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);          // writes to <topic>.DLT
    var handler = new DefaultErrorHandler(recoverer, new ExponentialBackOff(500, 2.0));   // 0.5 s, 1 s, 2 s ...
    handler.addNotRetryableExceptions(IllegalArgumentException.class);    // bad data will never succeed
    return handler;
}

Retrying inside the consumer blocks the partition: no later record is processed until the failing one succeeds or gives up. For a slow dependency that is the right behavior; for a poison record it is why a limit and a DLT matter. @RetryableTopic moves retries to separate topics so the main topic keeps flowing.

A runnable model of the container

The program below builds a miniature listener container with a consumer thread over an in-memory "partition". It has a registry that finds containers by id, start, pause, resume and stop, an error handler that retries a failing record twice, and a dead-letter list. Read the output top to bottom and match each line to a lifecycle call.

Java A listener container with a registry, pause/resume and a dead-letter topic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

A record that always fails is tried three times and then parked in the dead-letter list, so loan-3 was not stuck behind it forever. While the container was paused, new records stayed in the partition and were consumed after resume(), in order.

Kafka and MySQL with Docker Compose

For local development you rarely install Kafka by hand. A Compose file starts a single-node broker (modern Kafka runs without ZooKeeper, in KRaft mode) and a database:

YAML
services:
  kafka:
    image: apache/kafka:4.0.0
    ports:
      - "9092:9092"                     # the image's default listener is reachable at localhost:9092

  mysql:
    image: mysql:8.4
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: library
    ports:
      - "3306:3306"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-uroot", "-pexample"]
      interval: 5s
      retries: 10

Start it with docker compose up -d, then point the application at the published ports:

YAML
spring:
  kafka:
    bootstrap-servers: localhost:9092
  datasource:
    url: jdbc:mysql://localhost:3306/library
    username: root
    password: example

With the spring-boot-docker-compose dependency, Spring Boot starts the Compose file for you when the application starts and wires the connection details automatically. Two pitfalls cause most local problems. First, a Kafka broker advertises an address to its clients: an application inside the Compose network must use the service name (kafka:9092), while an application on your machine uses localhost:9092, and a broker advertising the wrong one makes the client connect and then time out. Second, container ports and data are temporary: use a named volume for the database if you want data to survive docker compose down. See Docker for Java and Testcontainers for repeatable environments in tests.

Delivery guarantees

Networks and processes fail at the worst moments. A consumer might process a message and crash before acknowledging it. A producer might send a message and lose the connection before the broker's confirmation arrives. Depending on how you handle this, you get one of three guarantees:

  • At most once: acknowledge (or commit) first, then process. A crash loses the message. Acceptable only for data you don't mind losing, such as some metrics.
  • At least once: process first, then acknowledge. A crash causes a redelivery, so the same message may be processed twice. This is the default and the right choice for almost everything.
  • Exactly once: processing takes effect once even with failures. Kafka offers this for read-process-write pipelines within Kafka (idempotent producers plus transactions). As soon as your consumer has side effects outside Kafka, such as writing to a database or sending an email, you are back to at-least-once delivery plus an idempotent consumer.

An idempotent consumer makes duplicates harmless. The usual technique: every message carries a unique eventId, and the consumer records processed ids in the same database transaction as its side effect. A duplicate finds its id already there and is skipped.

The next example simulates a broker with acknowledgments and redelivery. The consumer crashes once after doing its work but before the ack, so the broker redelivers. One message is poison: it fails every time, and after three attempts it is moved to a dead-letter queue instead of blocking the queue forever.

Java Redelivery, idempotent consumers and a dead-letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The naive consumer sent the reminder for loan 11 twice, because the redelivered message looked new. The idempotent consumer recognized evt-1 and skipped it. In both runs, the poison message evt-2 stopped being retried after three attempts and went to the dead-letter queue, where someone can inspect it, fix the bug or data, and replay it. Without a dead-letter queue, a single bad message can block a partition or spin in a retry loop forever.

Publishing reliably: the transactional outbox

The producer side has its own trap. A service saves a loan to its database and then publishes LoanCreated. If the broker is down, or the process crashes between the two steps, the database says the loan exists and no one else ever hears about it. Publishing first is no better: the event could go out for a loan that then fails to save.

The transactional outbox fixes this with the database you already have. In the same local transaction as the business change, insert the event into an outbox table. A separate relay reads unpublished rows, sends them to the broker and marks them as published. If the relay crashes after sending but before marking, the event is sent again, which is fine because consumers are idempotent.

Java A transactional outbox with H2

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The failed borrow left neither a loan nor an event, and each committed loan produced exactly one event. (The rolled-back insert still used up id 2: identity sequences never roll back, so gaps in ids are normal.) Running the relay twice didn't publish anything twice, because rows are marked. In production, the relay is either a scheduled job (as in Scheduling, Caching and Async Tasks) or a change data capture tool such as Debezium, which reads the database's transaction log and streams outbox rows to Kafka.

Designing messages

  • Events vs commands. An event states a fact in the past tense (LoanCreated, MemberBlocked); the producer doesn't care who listens. A command asks for something (SendReminder) and usually has one intended consumer. Prefer events between services; they keep coupling low.
  • Include metadata: a unique eventId, the event type, occurredAt, and a correlation or trace id. The CloudEvents specification standardizes these fields.
  • Thin or fat events. A thin event ({"loanId": 11}) forces consumers to call back for details; a fat event carries what consumers need (book id, member id, due date). Fat events decouple better but expose more of your data model.
  • Evolve schemas compatibly. Producers and consumers deploy at different times. Add optional fields, never rename or remove fields in place, and tolerate unknown fields on the consumer side (configure Jackson to ignore them; see Advanced Jackson). Avro or Protobuf with a schema registry enforce compatibility automatically.
  • Choose keys for ordering. In Kafka, key by the entity whose events must stay in order (bookId, orderId).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Three instances of the reminder service consume from one RabbitMQ queue. A message arrives. How many instances process it?

    A queue is point-to-point. To deliver to several services, bind one queue per service to an exchange.
  2. 2. A Kafka topic has 4 partitions. A consumer group has 6 consumers. What happens?

    Within a group, each partition goes to exactly one consumer, so the partition count caps the group's parallelism.
  3. 3. All events for the same book must be processed in order. What do you do in Kafka?

    Kafka orders records only within a partition, and the key selects the partition.
  4. 4. A consumer sends an email and then crashes before acknowledging the message. With at-least-once delivery, what happens and what protects you?

    Unacknowledged messages are redelivered. Kafka transactions don't cover external side effects like emails, so the consumer must deduplicate.
  5. 5. Why does the transactional outbox write the event to a database table instead of sending it to the broker directly?

    Two separate writes (database and broker) can't be atomic. The outbox turns them into one local transaction plus a retryable relay.

Summary

  • Messaging decouples services in time and load: producers publish to a broker, consumers process when ready.
  • Queues deliver each message to one competing consumer (work distribution); topics deliver to every subscriber (events).
  • RabbitMQ routes messages from exchanges (direct, topic, fanout) to queues via bindings; consumers ack, and acked messages are removed. Dead-letter exchanges catch failures.
  • Kafka stores records in partitioned, replayable logs. The key picks the partition, order holds per partition, and consumer groups split partitions among members while different groups read independently.
  • Plan for at-least-once delivery: process, then ack, and make consumers idempotent with unique event ids. Send poison messages to a dead-letter queue after a few attempts.
  • Publish reliably with the transactional outbox: write the event in the same transaction as the change, and relay it to the broker afterward.
  • Design events in the past tense with ids and timestamps, evolve them compatibly, and choose keys for ordering.

Next: Reactive Programming looks at streams of data inside a single service: publishers, subscribers and backpressure.