Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 37 ยท Communication

Asynchronous Messaging and Events

Decouple microservices with asynchronous messaging: queues vs publish/subscribe, commands vs events, at-least-once delivery, and a runnable in-memory broker.

With a synchronous call (Synchronous Communication with HTTP APIs) the caller waits for the callee. With asynchronous messaging the caller drops a message into a broker and carries on. The broker stores the message and delivers it to the receiver when the receiver is ready, even if that is a few seconds, or a few hours, later.

This removes the coupling in time: the sender does not need the receiver to be up, and a burst of traffic is absorbed by the broker instead of crushing the receiver. The price is new things to think about: ordering, duplicates, and the fact that the answer is no longer in the same call. This lesson explains the vocabulary and lets you run a small broker yourself.

Two shapes of messaging

Queue Each message goes to ONE worker.
Pub/sub Each message goes to EVERY subscriber.
1 / 2

Queues and publish/subscribe

A queue holds messages until a consumer takes one. If several consumers read the same queue, the messages are shared among them, so each message is handled by exactly one worker. This is how you spread work: image resizing, sending emails, generating reports.

Publish/subscribe (pub/sub) works differently. A publisher sends a message to a topic, and every subscriber gets its own copy. When orders publishes "order placed", billing, shipping and analytics each receive it independently.

Real brokers combine both: each subscribing service has its own queue fed by the topic, and the workers inside that service share that queue. RabbitMQ calls this exchanges and queues; Kafka calls it topics and consumer groups. Cloud brokers (SQS with SNS, Pub/Sub, Service Bus) have the same ideas under other names.

Commands and events

Not all messages mean the same thing.

  • A command is an instruction to a specific receiver: ReserveStock, ChargeCard. The sender knows who should act and expects it to happen.
  • An event is a statement of fact in the past tense: OrderPlaced, PaymentCaptured. The publisher does not know, or care, who listens.

Events give you loose coupling. When a new "loyalty points" service appears, it subscribes to OrderPlaced, and the orders service does not change at all. Name events from the business point of view (OrderPlaced), not by what a consumer should do (SendConfirmationEmail); the second is really a command in disguise.

Delivery guarantees

Networks and processes fail, so a broker must choose what to promise.

  • At-most-once. Deliver and forget. No duplicates, but messages can be lost.
  • At-least-once. Keep the message until the consumer acknowledges it. If the consumer crashes before acknowledging, the broker sends it again. Nothing is lost, but duplicates happen.
  • Exactly-once. Marketed often, but across systems it is really at-least-once delivery plus consumers that ignore duplicates. Treat "exactly once" as a goal you build, not a switch you flip.

At-least-once is the common default, so consumers must be idempotent: handling the same message twice must be harmless. The usual trick is to remember processed message ids. You will build this properly in The Transactional Outbox and Idempotent Consumers.

A broker you can run

The stand-in below is a tiny in-memory broker in one Node file. It supports topics, subscriber queues and acknowledgement: a message is removed from the queue only after the handler succeeds, and is redelivered when the handler fails. A real broker adds persistence to disk, networking and clustering, but the delivery logic is the same idea.

Node.js In-memory broker with acknowledgements

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three subscribers each received the event (pub/sub), and the failing one was retried until it worked while the others were not affected. The publisher finished instantly and never knew about any of them. Notice that billing got the message twice, once failed and once successful: that is at-least-once delivery in action.

Queues and competing consumers

For a queue with several workers, each message goes to one of them. That is called the competing consumers pattern, and it is how you scale a slow step by simply starting more workers.

Node.js Competing consumers share one queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Eight jobs took three rounds instead of eight, and no job was done twice. Because messages are spread across workers, ordering is only guaranteed per queue (or per partition in Kafka), never across workers. If two messages about the same order must be processed in order, send them to the same partition with a partition key such as the order id.

What messaging costs you

Asynchrony is powerful but not free.

  • Eventual consistency. After "order placed" is published, the invoice does not exist yet. User interfaces and APIs must cope with in-between states.
  • No immediate answer. To report results, use another message, a status endpoint, or a notification.
  • Debugging gets harder. A request becomes a trail across services. You will need correlation ids and tracing (Distributed Tracing).
  • Poison messages. A message that always fails would be retried forever. Brokers offer a dead-letter queue where such messages are parked for a human (Project: Event-Driven Notification System).
  • The broker is infrastructure you must run, monitor and secure.

Choosing a broker

The concepts above apply to every broker; the products differ in how they store and deliver messages.

Broker Model Strengths Typical uses
Apache Kafka Distributed, partitioned, replicated log; consumers track offsets and can replay Very high throughput, long retention, replay, stream processing Event streams, analytics feeds, event sourcing, IoT telemetry
RabbitMQ Exchanges route to queues (direct, topic, fanout); broker tracks acknowledgements Flexible routing, per-message acks, priorities, low latency Task queues, work distribution, request/reply
Apache ActiveMQ JMS broker (Classic) and a newer high-throughput variant Standard Java messaging API, multiple protocols (AMQP, MQTT, STOMP) Enterprise integration, Java shops
Azure Service Bus / AWS SQS+SNS / Google Pub/Sub Managed queues and topics No servers to run, built-in scaling and dead-letter queues Cloud-native services that want low operations effort

A message broker also does protocol transformation (an MQTT device can feed an AMQP consumer) and decouples producers from consumers, which is why it shows up in telecommunications, healthcare integration (hospital systems exchanging records), IoT and order processing alike. To choose: ask whether consumers need to replay history (log-based like Kafka), whether you need rich routing and per-message handling (RabbitMQ-style), and how much operational work you can take on (managed services). Start with the simplest thing that meets the delivery guarantees you need; the system-design view of the same decision is in Message Queues and Pub/Sub and Brokers.

Real broker configuration

These snippets are real configuration, read-only because the site cannot run a broker. First, a Kafka topic with three partitions, created with the command line tool:

Terminal
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic order.placed \
  --partitions 3 --replication-factor 3

And a RabbitMQ setup in definitions format: one topic exchange, one queue for billing, and a binding between them.

JSON
{
  "exchanges": [
    { "name": "orders", "type": "topic", "durable": true }
  ],
  "queues": [
    { "name": "billing.order-placed", "durable": true }
  ],
  "bindings": [
    {
      "source": "orders",
      "destination": "billing.order-placed",
      "destination_type": "queue",
      "routing_key": "order.placed"
    }
  ]
}

Choose a broker after you know the need. Kafka keeps a replayable log and handles huge volumes (useful for Event Sourcing and CQRS); RabbitMQ and cloud queues excel at task distribution with flexible routing.

Keep event payloads small and stable, and include an event id, a timestamp and a version. Consumers will depend on every field you send.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main difference between a queue and pub/sub?

    A queue spreads work among consumers; pub/sub fans a message out to everyone who subscribed.
  2. 2. Which is an event, rather than a command?

    Events are facts in the past tense. The publisher does not tell anyone what to do.
  3. 3. With at-least-once delivery, what must consumers be ready for?

    If an acknowledgement is lost or the consumer crashes after working, the broker redelivers, so the same message may be handled twice.
  4. 4. How do you keep all messages about one order in order when using several partitions?

    Messages with the same key go to the same partition, and order is preserved within a partition.
  5. 5. What is a dead-letter queue for?

    After repeated failures a message is moved aside for inspection, instead of being retried endlessly.

Summary

  • Asynchronous messaging decouples sender and receiver in time: the broker stores messages until the consumer is ready.
  • Queues give each message to one worker; pub/sub gives every subscriber a copy.
  • Commands ask a service to act; events state facts that anyone may react to.
  • At-least-once delivery is the practical default, so consumers must be idempotent.
  • The costs are eventual consistency, harder debugging, poison messages and one more system to run.

Next: RPC, gRPC and Protocol Buffers.