Skip to content
Tutorials Center
My learning

Topics

Lesson 112 of 133 ยท Production Node.js

Microservices and Message Queues

Learn service boundaries, message queues, at-least-once delivery, retries with backoff, dead-letter queues and idempotent consumers, with runnable Node.js examples.

When an application grows, one codebase and one process can become hard to change and hard to scale. Some teams respond by splitting the system into several small programs that each own one job and talk to each other over the network. These are microservices. The idea is attractive, but a network is slower and less reliable than a function call, so the split brings a new set of problems: what happens when a call times out, when a message arrives twice, or when the receiving service is down for a minute?

This lesson focuses on the parts that matter most for a Node.js developer: where to draw service boundaries, how message queues decouple services, and the three habits that make distributed code survive real life: retries, idempotency and dead-letter queues. Everything runs inside one process with the built-in tools, so you can watch the failures happen. For the single-process groundwork, see Scalability Patterns, Graceful Shutdown and Health Checks and the Background Job Queue project.

Service boundaries

A good boundary follows a business capability, not a technical layer. "Orders", "Payments" and "Inventory" are services; "Database service" and "Validation service" are not. A few guidelines:

  • Each service owns its data. Other services never read its tables; they ask through its API or listen to its events. Shared databases hide coupling and make every change risky.
  • A service should be changeable and deployable without deploying others. If two services always change together, they are really one service.
  • Prefer fewer, larger services at first. Splitting later is easy; merging a hundred chatty services is not. A well-structured single application (a "modular monolith") is a perfectly good place to start, and most products never need more.

Services talk in two styles:

Style Example Good for Cost
Synchronous request/response HTTP or gRPC call Needing the answer now (a price lookup) Caller fails when the callee is down or slow
Asynchronous messages Publish to a queue or topic Work that can happen later (send an email, update a report) Eventual consistency, duplicates, ordering questions
flowchart LR
  C["Client"] --> O["Orders service"]
  O -->|"publishes order.placed"| Q[("Message queue")]
  Q --> P["Payments service"]
  Q --> I["Inventory service"]
  Q --> N["Notifications service"]
  P --> OD[("Payments DB")]
  I --> ID[("Inventory DB")]

Orders publishes one event and does not know who listens; each consumer has its own database.

What a message queue gives you

A message queue (RabbitMQ, Amazon SQS, Redis streams, Kafka-style logs and others) is a middleman that stores messages until a consumer takes them. It gives you:

  • Decoupling. The producer does not wait for, or even know about, the consumer.
  • Buffering. A traffic spike turns into a longer queue instead of a crash; consumers work at their own speed.
  • Retry and recovery. If a consumer dies, the message is still there.
  • Scaling. Add more consumers to read the same queue in parallel.

Most queues promise at-least-once delivery: a message is handed to a consumer and stays reserved; if the consumer does not acknowledge (ack) it in time, the broker hands it out again. That guarantees nothing is lost, but it means duplicates will happen: the consumer may finish the work and crash before the ack, or the ack may be lost on the network. "Exactly once" is not something you can simply switch on; you build the effect yourself using idempotency, shown below.

sequenceDiagram
  participant P as Producer
  participant Q as Queue
  participant C as Consumer
  P->>Q: publish(message)
  Q->>C: deliver(message)
  C->>C: do the work
  C-->>Q: ack
  Note over Q,C: No ack in time? The queue delivers it again

The delivery cycle: a message is only removed after the consumer acknowledges it.

A tiny in-memory queue

Real brokers need servers, so we build the smallest queue that shows the contract: publish, a consumer that must ack or nack, and a limit on delivery attempts. After too many failures the message moves to a dead-letter queue (DLQ), a parking place where a human or a repair job can inspect poison messages instead of letting them block everything.

Retries need a delay that grows, called exponential backoff: wait 100 ms, then 200, then 400. Adding randomness (jitter) stops many clients from retrying at the same moment. To keep the output stable, the demo records the delays it would wait instead of sleeping, and uses a fixed jitter function.

Node.js A queue with retries, backoff and a dead-letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two things to carry over to real systems. First, only retry errors that can succeed later (timeouts, 503, connection resets); a validation error will fail forever, so send it to the DLQ immediately. Second, cap the retries. An unlimited retry loop against a failing dependency makes the outage worse.

Idempotency: making duplicates harmless

An operation is idempotent when doing it twice has the same effect as doing it once. Setting a field is idempotent; "add 10 to the balance" is not. Because queues deliver at least once and clients retry on timeouts, every handler that changes something must either be naturally idempotent or defend itself.

The standard defense is a deduplication key: every message (or request) carries a unique ID, and the consumer remembers the IDs it has already processed. If the ID is known, it skips the work and acknowledges again. The check and the work must happen atomically in real systems: with a database, insert the ID and update the balance in one transaction (a unique constraint on the ID makes the second insert fail). The same idea appears in HTTP as an Idempotency-Key header, which you met in Designing REST APIs.

The demo delivers the same payment message twice, as an at-least-once broker may. The naive consumer charges twice; the idempotent one does not.

Node.js Duplicate delivery: naive vs idempotent consumer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Alice paid 35 in total, but the naive consumer shows 65. Keep the processed IDs for as long as duplicates can still arrive (for a queue, longer than its maximum redelivery window); a TTL of days is common.

Synchronous calls between services

Not everything is a queue. When Orders must call Payments over HTTP, protect the call:

  • Set a timeout on every outgoing request (AbortSignal.timeout(2000) works with fetch; see Abort Signals and Cancellation). A missing timeout turns one slow service into a stuck chain of waiting callers.
  • Retry only safe or idempotent calls, with backoff and jitter, and a maximum.
  • Send an idempotency key with every unsafe call, so a retry after a lost response cannot charge the customer twice.
  • Consider a circuit breaker: after several failures, stop calling the broken service for a while and fail fast, then probe again. It protects the callee from a retry storm and the caller from waiting.

The next example shows the idempotency key in action. The payments service is deliberately unreliable: it does the work, then "loses" the response of the first attempt (we simulate it by destroying the connection). The client retries with the same key and gets the stored result instead of a second charge.

Node.js Retrying a payment safely with an idempotency key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The client saw an error on attempt 1, yet the customer was charged exactly once. Without the key, the retry would have created a second charge.

Events, ordering and consistency

A few more realities of asynchronous systems:

  • Ordering. Most queues only guarantee order inside one partition or one queue with one consumer. If two messages for the same order must be processed in order, route them by order ID to the same partition, or make the handler tolerate any order (for example by storing a version number and ignoring older events).
  • Eventual consistency. After "order placed", the inventory count changes a moment later. Design the user interface and APIs for that (show "processing").
  • No distributed transactions. You cannot wrap a payment, a stock update and an email in one database transaction across services. Use a saga: a sequence of local steps, each with a compensating action (refund, restock) that runs if a later step fails.
  • The dual-write problem. "Save the order to the database, then publish the event" can fail between the two steps. The outbox pattern writes the event into an outbox table in the same transaction as the order, and a separate relay publishes it, giving at-least-once publishing.
  • Observability. Put a correlation ID in every message and log line so you can follow one request across services (see Logging and AsyncLocalStorage).

When you run each service as a container, health checks and graceful shutdown (see Graceful Shutdown and Health Checks) are what let a deploy replace consumers without losing in-flight messages: stop receiving, finish the current message, ack, exit. The next lesson, Node.js in Docker and CI, shows how such services are packaged.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does "at-least-once delivery" mean for a consumer?

    If the ack is lost or the consumer crashes after doing the work, the broker redelivers the message. Nothing is lost, but duplicates happen, so handlers need to be idempotent.
  2. 2. Which service split follows good boundary rules?

    Boundaries follow business capabilities, and each service owns its data. Shared tables or purely technical services create tight coupling.
  3. 3. What is a dead-letter queue for?

    After the maximum number of attempts a message is parked in the DLQ so it can be inspected or repaired, instead of being retried forever.
  4. 4. Why add jitter to exponential backoff?

    Without randomness, clients that failed together retry together and can overload a recovering service again. Jitter spreads the retries out.
  5. 5. A client retries a POST /charge after a timeout. How do you prevent a double charge?

    The server saves the outcome under the key. A repeated request with the same key returns the stored result without performing the charge again.

Summary

  • Draw service boundaries around business capabilities, let each service own its data, and start with fewer, larger services.
  • Use synchronous calls when you need an answer now, and queues for work that can happen later; queues decouple, buffer and allow retries.
  • Brokers usually give at-least-once delivery, so duplicates are normal: acknowledge only after the work is done.
  • Retry only errors that can recover, with exponential backoff, jitter and a cap; park poison messages in a dead-letter queue.
  • Make handlers idempotent with a deduplication key stored atomically with the work; send Idempotency-Key on unsafe HTTP calls.
  • Always set timeouts, consider circuit breakers, and use sagas and the outbox pattern instead of distributed transactions.
  • Carry a correlation ID through every message for tracing.

Next: Node.js in Docker and CI