Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 66 ยท Projects

Project: Event-Driven Notifications

Build reliable event-driven notifications in Spring Boot: domain events, the outbox pattern, a relay, a retrying listener, idempotency, dead letters and metrics.

When an order is placed, you want to tell the customer. The obvious code saves the order and then calls the mail service in the next line. It works until the day the mail server is slow, the application crashes between the two lines, or the database commit fails after the email was already sent. Then customers get emails for orders that do not exist, or never get an email for orders that do. Two systems cannot be updated in one transaction, so you need a design that is reliable even when parts fail.

This project builds that design in four pieces: a domain event, an outbox that records the event in the same transaction as the business change, a relay that delivers it, and a listener that retries failures and ignores duplicates. Spring Boot, Kafka and a real mail server cannot run on this site, so the Spring code is read-only and the whole pipeline is rebuilt live in plain Java with an in-memory outbox and broker. The stand-ins say plainly what they replace. It continues the use case of Project: Bookstore Orders Service, whose AFTER_COMMIT listener is the simple version of what we make robust here.

The reliable pipeline

Save both Order and outbox row in one transaction.
Relay A poller sends pending rows to the broker.
Listener Handles each message, retries on failure.
Idempotent A duplicate message changes nothing.
1 / 2

Why not just publish inside the transaction?

There are three naive designs, and each fails in a different way:

  1. Send, then commit. If the commit fails, the message is already out. The world has been told about something that never happened.
  2. Commit, then send. If the application crashes between the two steps, the message is lost forever. Nothing records that it was still owed.
  3. Send inside a transaction-bound listener. @TransactionalEventListener(AFTER_COMMIT) fixes the first problem, but has the same hole as design 2: a crash after the commit loses the event.

The transactional outbox removes the gap. Instead of sending the message, the business transaction writes a row to an outbox table in the same database, in the same commit. Either both the order and the "to be announced" row exist, or neither does. A separate process then reads pending rows, sends them and marks them as sent. The delivery can be retried as often as needed, because the row is still there until it is confirmed. For the broader messaging picture, see Messaging and Events Between Services.

flowchart LR
    A["Service<br/>@Transactional"] -->|"one commit"| B[("orders +<br/>outbox table")]
    B --> C["Relay<br/>(scheduled poller)"]
    C -->|"send"| D["Broker"]
    D --> E["Listener<br/>(idempotent)"]
    E --> F["Mail service"]

The only step that must be atomic is the first one; everything after it can fail and be repeated.

The pieces in Spring Boot

The event is a small immutable record with a unique id. The id is what makes duplicate detection possible later:

Java
public record OrderPlaced(UUID eventId, long orderId, String email, int totalCents) {}

The outbox row stores the event as JSON, plus bookkeeping for the relay:

SQL
create table outbox (
    id           uuid         primary key,
    type         varchar(100) not null,
    payload      text         not null,
    status       varchar(20)  not null default 'PENDING',
    created_at   timestamp    not null default current_timestamp,
    sent_at      timestamp
);
create index idx_outbox_pending on outbox (created_at) where status = 'PENDING';

The service writes the order and the outbox row together. There is no call to the broker here:

Java
@Service
public class OrderService {

    private final OrderRepository orders;
    private final OutboxRepository outbox;
    private final ObjectMapper json;

    // constructor omitted

    @Transactional
    public CustomerOrder placeOrder(PlaceOrderRequest request) throws JsonProcessingException {
        CustomerOrder order = orders.save(CustomerOrder.from(request));
        var event = new OrderPlaced(UUID.randomUUID(), order.getId(),
                order.getCustomerEmail(), order.getTotalCents());
        outbox.save(new OutboxMessage(event.eventId(), "OrderPlaced", json.writeValueAsString(event)));
        return order;
    }
}

The relay is a scheduled job. It claims a batch of pending rows, publishes each and marks it sent. With several application instances running, FOR UPDATE SKIP LOCKED lets each instance claim different rows instead of blocking or sending the same row twice:

Java
@Component
public class OutboxRelay {

    private final OutboxRepository outbox;
    private final KafkaTemplate<String, String> kafka;

    // constructor omitted

    @Scheduled(fixedDelay = 1000)
    @Transactional
    public void relay() {
        for (OutboxMessage message : outbox.claimPending(50)) {
            kafka.send("orders.events", message.getId().toString(), message.getPayload()).join();
            message.markSent();
        }
    }
}

public interface OutboxRepository extends JpaRepository<OutboxMessage, UUID> {
    @Query(value = """
            select * from outbox where status = 'PENDING'
            order by created_at limit :max for update skip locked
            """, nativeQuery = true)
    List<OutboxMessage> claimPending(@Param("max") int max);
}

Notice the failure window: if the application crashes after Kafka accepted the message but before the transaction marks the row as sent, the next run sends the same message again. The outbox gives you at-least-once delivery, never exactly once. That is not a bug in the design; it is the reason the listener must be idempotent. Scheduling itself is covered in Scheduling and Asynchronous Work.

Run the pipeline

The first example is the whole path in memory. placeOrder appends to an orders list and an outbox list inside one synchronized block (the stand-in for a database transaction). The relay marks rows as sent only after the "broker" accepted them, and we make it crash once in the dangerous window. The consumer keeps a set of processed event ids, like a processed_message table with a unique key.

Java Outbox, relay and an idempotent consumer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Three messages crossed the topic, but only two emails went out. The crash produced a duplicate, the outbox made sure nothing was lost, and the idempotent listener made the duplicate harmless. In real code, the "already handled" check is an insert into processed_message (event_id) whose unique constraint throws on a repeat. Do it in the same transaction as the work it protects, so a failure does not leave the id recorded without the email being sent (or the other way round).

The listener: retries, backoff and dead letters

Delivery to the listener succeeded, but the listener itself can fail: the mail server times out, or an address is permanently invalid. Those two failures need different treatment. A transient failure deserves another attempt after a pause that grows (exponential backoff), so you do not hammer a struggling server. A permanent failure will never succeed, so after a limited number of tries the message goes to a dead-letter place where people can inspect and replay it, instead of blocking everything behind it.

Java
@Component
public class NotificationListener {

    private static final Logger log = LoggerFactory.getLogger(NotificationListener.class);

    private final ProcessedMessages processed;
    private final MailService mail;
    private final MeterRegistry metrics;

    // constructor omitted

    @KafkaListener(topics = "orders.events", groupId = "notifications")
    @Transactional
    public void onMessage(String payload) throws JsonProcessingException {
        OrderPlaced event = json.readValue(payload, OrderPlaced.class);
        if (!processed.markIfNew(event.eventId())) {
            metrics.counter("notifications.duplicate").increment();
            return;
        }
        mail.sendConfirmation(event.email(), event.orderId());   // may throw
        metrics.counter("notifications.sent", "channel", "email").increment();
    }
}

With Spring for Apache Kafka, retries and the dead-letter topic are configured on the container's error handler rather than written by hand: a DefaultErrorHandler with an exponential BackOff and a DeadLetterPublishingRecoverer. Spring Framework 7 also has its own @Retryable annotation for plain method calls; both are explained in Resilience: Timeouts, Retries and Circuit Breakers. Retried work must stay idempotent, which is exactly why the duplicate check comes first.

The second example runs the retry policy with a simulated clock, so it finishes instantly and always prints the same output. The mail sender fails twice for one customer, always for another, and succeeds for the rest. Counters stand in for Micrometer meters.

Java Retry with backoff, dead letters and counters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Bob's second retry waited 200 ms and then 400 ms before succeeding, Eve was dead-lettered at once because retrying a nonexistent mailbox can never help, and every outcome moved a counter. A dashboard that shows notifications.exhausted rising is the alarm that tells you the mail provider is down.

Observability: seeing the pipeline

An asynchronous pipeline fails quietly: the request that caused the problem finished minutes ago. Add three kinds of signals, the ones introduced in Observability: Metrics, Tracing and Logs:

  • Counters for sent, duplicate, retried and dead-lettered messages, as in the example, plus a gauge for the number of pending outbox rows and the age of the oldest one. A growing backlog means the relay or broker is stuck, even when nothing throws.
  • Logs with a correlation id. Put the eventId into the logging context (MDC) when the listener starts, so one search finds the relay line, the listener line and the retry lines of the same event.
  • Alerts on symptoms, not causes: "oldest pending outbox row is older than five minutes" and "dead-letter count increased" both catch problems you did not predict.
Java
Gauge.builder("outbox.pending", outbox, OutboxRepository::countPending)
     .description("Messages waiting to be sent")
     .register(metrics);

Extending the project

  • Add a cleanup job that deletes outbox rows with status = 'SENT' older than seven days, so the table stays small.
  • Add a second listener (for example "decrease analytics counters") to prove that one event can feed several independent consumers, each with its own processed-id table.
  • Add an admin endpoint that lists dead letters and replays one by publishing it again; keep it behind authentication (see Spring Security Fundamentals).
  • Write a test that kills the relay between "send" and "mark sent" and asserts that exactly one email goes out, like the first live example.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does the transactional outbox solve?

    Writing the order and the outbox row in one transaction means both exist or neither does. Delivery is then retried until it is confirmed.
  2. 2. Why must the listener be idempotent in this design?

    At-least-once delivery is the price of never losing a message. The consumer must treat a repeated event id as already done.
  3. 3. A message fails because the recipient's mailbox does not exist. What is the best handling?

    A permanent failure never succeeds on retry, so retrying wastes time and delays other messages. A dead letter keeps the message for inspection and replay.
  4. 4. Which signal best reveals that the relay is stuck even though no exception is thrown?

    Silent stalls show up as a growing backlog. An age-based alert catches them whatever the cause.
  5. 5. Why does FOR UPDATE SKIP LOCKED help in the relay query?

    Locked rows are skipped, so each instance works on its own batch. The processed-id check still protects against the rare duplicates.

Summary

  • Updating a database and a broker cannot be atomic, so write the event to an outbox table in the same transaction as the business change.
  • A relay sends pending rows and marks them as sent; a crash in between causes a duplicate, never a lost message (at-least-once delivery).
  • Make consumers idempotent with a unique-key table of processed event ids, checked in the same transaction as the work.
  • Retry transient failures with exponential backoff, dead-letter permanent ones or exhausted ones, and keep the dead letters replayable.
  • Observe the pipeline with counters, a backlog gauge, correlation ids in logs and alerts on symptoms.

Next: Practice Exercises with Solutions.