Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Real-World Projects

Project: Event-Driven Notification System

Build a notification system with producers, a broker, fan-out consumers, retries with backoff, a dead-letter queue and redrive, in runnable Node code.

Almost every product sends notifications: "your order has shipped", "here is your password reset link", "your invoice is overdue". They are the perfect microservices project, because the work is naturally asynchronous (nobody should wait for an email provider while paying) and naturally failure-prone (email, SMS and push services time out, rate-limit and reject things).

In this project you build the heart of such a system. Business services (producers) only announce what happened. A router decides who should hear about it and how. One consumer per channel (email, SMS, push) does the delivery, retries what can be retried, and parks what cannot in a dead-letter queue until a human or a script fixes the cause. The broker here is an in-memory stand-in that follows the rules of real brokers (Asynchronous Messaging and Events).

Architecture

flowchart LR
    A["Orders service"] -->|order.shipped| B(("Broker"))
    C["Auth service"] -->|password.reset| B
    B --> R["Router"]
    R -->|send.email| B
    R -->|send.sms| B
    R -->|send.push| B
    B --> E["Email sender"]
    B --> S["SMS sender"]
    B --> P["Push sender"]
    E -.->|gives up| D[("Dead-letter queue")]
    S -.->|gives up| D
    P -.->|gives up| D

Producers publish facts, the router fans each one out per channel, and senders that give up park the message in the dead-letter queue.

Design rules

Producers announce They publish events and never call senders.
One router Preferences and templates live in one place.
One consumer per channel Each fails and scales on its own.
Dead-letter queue Nothing is silently lost.
1 / 2

Step 1: the routing decision

The first design question is what to send, to whom, through which channel. Keep that logic in one small, pure, easily testable function: given an event, the user's preferences and the time, return a plan. The stand-in below includes user channel preferences, quiet hours (no SMS or push at night) and a critical flag that overrides quiet hours for things like password resets.

Node.js The routing plan: preferences, templates and quiet hours

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two details are worth noticing. Email is treated as non-intrusive, so it is always sent and waits in the inbox. And the same function decides about deferral; in a real system the "defer" action would schedule the message for 07:00 (a delayed message in the broker, or a row in a scheduled-jobs table). A pure function like plan is exactly what you would cover with unit tests (Testing Microservices).

Step 2: broker, consumers, retries and dead letters

Now the full machine. Read it in three blocks: the Broker class (queues per consumer, exponential backoff, a dead-letter queue and redrive), the router and channel consumers, and the scenario at the end. The broker uses a virtual clock, so the "waiting" for a retry costs no real time and the output is the same on every run.

Node.js Fan-out, retries with backoff, dead letters and redrive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Follow the story in the output:

  • The duplicate evt-1 was caught by the router's memory of processed ids and ignored, so Ada did not get two emails. (The router logs nothing when it works, only when it skips.)
  • Bob's SMS failed twice with a transient error. The broker waited 2 seconds, then 4 (exponential backoff, Timeouts, Retries and Exponential Backoff) and the third attempt at t=6 worked. While it waited, the other messages were delivered; one slow message never blocked the rest.
  • Carl's push failed with a permanent error. Retrying a missing token is pointless, so it went straight to the dead-letter queue instead of burning attempts.
  • After Carl registered a token, redrive() moved the parked message back and it was delivered. Ids were kept, so even if an operator redrove twice, the once wrapper would drop the second delivery.

Design decisions you should be able to defend

Why one topic per channel? Each channel has a different failure profile and capacity. If SMS is down, email must keep flowing. Separate queues are bulkheads (Circuit Breakers, Bulkheads and Fallbacks).

Why mark success only after the work? The once helper records the message id after the send. If the process crashed between sending and recording, the message would be sent again, which is the at-least-once trade-off: a rare duplicate is better than a lost notification. For stronger guarantees, store the id and the effect in one transaction (The Transactional Outbox and Idempotent Consumers).

Why a dead-letter queue and not infinite retries? A poison message would otherwise block capacity forever. The DLQ needs an owner: an alert when it is non-empty, a dashboard, and a documented way to inspect and redrive.

Why do producers publish events instead of "send an email" commands? Because the orders team should not know that emails, SMS and push exist. When marketing adds a Slack channel, only the notification service changes.

Extending the project

  1. Rate limiting per provider. SMS vendors throttle; add a token bucket in the SMS consumer (Rate Limiting, Load Shedding and Backpressure).
  2. Add jitter to the backoff so thousands of failed messages do not retry at the same moment.
  3. Priority queues. Give password resets their own queue so a marketing blast cannot delay them.
  4. Delivery receipts. Emit NotificationDelivered and NotificationFailed events and build a status page from them.
  5. Metrics. Count sends, retries and dead letters per channel, and alert on the DLQ size (Observability: Structured Logs and Metrics).

What is real here: the pattern (fan-out, per-consumer queues, backoff, DLQ, idempotent consumers, redrive). What is simulated: the broker lives in memory, time is virtual, and the senders only print. Real systems use RabbitMQ, SQS, Kafka or similar with persistence, and real provider SDKs.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do the producers publish events such as order.shipped rather than calling an email service?

    Events express facts. The notification service decides what to do with them, which keeps producers decoupled.
  2. 2. Why does a missing push token go straight to the dead-letter queue?

    Retries help with transient problems like timeouts. A permanent error needs a human or script to fix the cause, then a redrive.
  3. 3. What does the exponential backoff in the example achieve?

    Waiting 2s, 4s, 8s and so on spreads load and gives transient failures time to clear.
  4. 4. Why does each consumer remember processed message ids?

    Duplicates come from retries, redrives and producer re-publishing. Idempotent consumers make them harmless.
  5. 5. What must exist around a dead-letter queue to make it useful?

    An unwatched DLQ is just a place where failures go to be forgotten.

Summary

  • Producers publish facts; a router turns each fact into per-channel messages using preferences, templates and quiet hours.
  • One queue and consumer per channel isolates failures and lets each channel scale independently.
  • Retry transient errors with exponential backoff; send permanent errors and exhausted messages to a dead-letter queue.
  • Consumers are idempotent because delivery is at-least-once, and redriving the DLQ is safe because of it.
  • Watch the dead-letter queue: it is a to-do list, not a trash can.

Next: Project: Scalable URL Shortener.