Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Patterns: RPC, Work Queues

Publish/Subscribe and Event Fanout

Design event-driven systems with RabbitMQ publish/subscribe: one queue per subscribing service, fanout and topic exchanges, competing consumers per service, event naming and the outbox pattern.

In a work queue each message is handled by one worker. In publish/subscribe (pub/sub) a message is an event, "order 42 was placed", and every interested service gets its own copy: billing charges the card, the warehouse reserves stock, the email service sends a confirmation. The publisher does not know who listens. New subscribers can be added without touching it.

RabbitMQ implements pub/sub with exchanges and queues rather than topics. The golden rule: one queue per subscribing service, bound to the event exchange. Inside a service, several instances share that queue as competing consumers.

RabbitMQ cannot run on this page, so the pika code is read-only. The runnable blocks are plain Python stand-ins for exchange routing, so you can see which queue receives what. They are not the broker.

Anatomy of an event system

flowchart LR
    O[Order service] -->|order.placed| X((events<br/>topic exchange))
    X -->|order.*| QB[billing.orders]
    X -->|order.placed| QW[warehouse.orders]
    X -->|"#"| QA[audit.all]
    QB --> B1[billing instance 1]
    QB --> B2[billing instance 2]
    QW --> W1[warehouse]
    QA --> A1[audit]
  • The publisher writes to a topic exchange named events with a routing key such as order.placed.
  • Each subscriber declares its own durable queue and binds it with the patterns it cares about (order.*, order.placed, #).
  • Two instances of billing consume the same billing.orders queue and split the work. A new feature that needs the same events (say, a loyalty service) declares its own queue; nothing existing changes.
Python
# publisher (order service)
channel.exchange_declare(exchange="events", exchange_type="topic", durable=True)
channel.basic_publish(
    exchange="events",
    routing_key="order.placed",
    body=json.dumps({"order_id": 42, "total": 59.90}).encode(),
    properties=pika.BasicProperties(
        delivery_mode=pika.DeliveryMode.Persistent,
        content_type="application/json",
        type="order.placed",
        message_id="evt-0001",
    ),
)

# subscriber (billing service), run once at startup
channel.exchange_declare(exchange="events", exchange_type="topic", durable=True)
channel.queue_declare(queue="billing.orders", durable=True,
                      arguments={"x-queue-type": "quorum"})
channel.queue_bind(queue="billing.orders", exchange="events", routing_key="order.*")

Both sides declare the exchange; declaring an existing entity with identical arguments is harmless, and it removes startup-order problems.

Fanout versus topic exchanges

A fanout exchange ignores the routing key and copies the message to every bound queue: simplest for "broadcast everything". A topic exchange matches the routing key against patterns: * replaces exactly one word and # zero or more words, with words separated by dots. For events, topic is almost always the better choice: you can start with # and narrow later. A direct exchange suits exact keys.

Python Stand-in: topic routing to service queues

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that shipment.sent reaches only audit.all, and the * in order.* matches one word, so refund.issued.partial needs refund.#.

Competing consumers inside a service

Each subscriber queue behaves like a work queue. The next block shows two services, with billing running two instances.

Python Stand-in: fanout to services, round-robin inside

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each event is handled once per service, never twice by the same service, as long as consumers ack properly.

Temporary subscribers

A subscriber that only wants live events, such as a dashboard, can declare an exclusive, server-named, auto-delete queue:

Python
result = channel.queue_declare(queue="", exclusive=True)
channel.queue_bind(exchange="events", queue=result.method.queue, routing_key="order.#")

The queue disappears when the dashboard disconnects, and events published while it was away are not stored. Real services need a durable named queue so they do not miss events during a deployment. For replay and many independent readers of the same history, consider a stream instead.

Designing events

  • Name by what happened, in the past tense: order.placed, payment.captured. Commands ("charge this card") belong on work queues.
  • Keep keys hierarchical (domain.entity.action), so topic patterns stay useful.
  • Version payloads: add a type header or schema_version field; consumers ignore unknown fields.
  • Put an id on every event (message_id) and let consumers deduplicate, as shown in Work Queues and Fair Dispatch.
  • Do not leak queues between services. Each service owns its queue and bindings; publishers own the exchange.

The outbox pattern

A service often has to update its database and publish an event. If it commits and then crashes before publishing, the event is lost; if it publishes first and the commit fails, the event lies. The transactional outbox fixes this: in the same database transaction that changes the data, insert the event into an outbox table. A separate relay process reads unsent rows, publishes them with confirms, and marks them sent. Delivery is at-least-once, so subscribers deduplicate by id.

Python Stand-in: outbox relay with retries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do three different services each receive every order.placed event?

    An exchange copies a message into every matching bound queue. One queue per service gives each its own copy.
  2. 2. What does the pattern order.* match?

    • replaces exactly one word. Use order.# to match order.placed.eu too.
  3. 3. Why is a durable named queue better than an exclusive temporary queue for a billing service?

    A temporary queue vanishes with its consumer, so events during downtime are lost.
  4. 4. What problem does the transactional outbox solve?

    The event is stored in the same transaction as the data and published afterwards by a relay, so neither is lost or invented.

Summary

  • Pub/sub delivers each event to every interested service: one durable queue per service, bound to a topic (or fanout) exchange.
  • Instances of the same service compete on one queue; topic patterns use * for one word and # for many.
  • Name events in the past tense, version payloads and add ids so consumers can deduplicate.
  • The outbox pattern makes database changes and published events consistent.

Next: Management UI, HTTP API and CLI