Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 65 ยท Asynchronous Systems

Publish/Subscribe and Message Brokers

Learn the publish/subscribe pattern and message brokers: topics, subscriptions, fan-out, consumer groups, RabbitMQ versus Kafka and delivery trade-offs, with Python.

A message queue hands each message to one worker. But often one event matters to many parts of a system: when an order is placed, billing must charge, inventory must reserve, email must confirm and analytics must count. With a queue, the order service would need to know about every one of them. Publish/subscribe (pub/sub) removes that knowledge. The publisher announces an event to a topic, and every subscriber interested in that topic receives its own copy.

This lesson explains topics, subscriptions, consumer groups and the broker that makes it all work, and compares the main styles of broker.

Queue versus pub/sub

One message, who gets it?

Queue One worker gets each message.
Pub/Sub Every subscriber gets a copy.
Consumer group Copies per group, one worker inside it.

Consumer groups combine both ideas.

1 / 2

The most useful model combines the two. Each service has its own subscription (a consumer group) to a topic. Every group receives every message, but inside a group, only one instance handles it. So billing gets each order once, and so does inventory, and each can run several instances to scale.

flowchart LR
  OS[Order service] -->|publish OrderPlaced| T{{Topic: orders}}
  T --> G1[Billing group]
  T --> G2[Inventory group]
  T --> G3[Email group]
  G1 --> B1[Instance 1]
  G1 --> B2[Instance 2]

The topic fans out to each group; instances inside a group share its copy.

The broker

The message broker is the server in the middle. It accepts messages, stores them until delivered, tracks who has seen what and routes by topic. Its job is to make producers and consumers independent in three ways:

  • Time: a subscriber can be offline; the broker holds messages.
  • Space: publishers do not know subscriber addresses.
  • Speed: slow consumers do not slow publishers.
Python A tiny broker with topics and consumer groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line shows a design choice: a classic broker only delivers messages published after you subscribe. Log-based brokers such as Kafka keep history, so a new subscriber can start from the beginning (see Event Sourcing and Streaming).

Topic routing and filtering

Subscribers rarely want everything. Brokers offer ways to select:

  • Topics by name: orders, payments. Simple and coarse.
  • Routing keys or wildcards: orders.eu.created, subscribed with orders.eu.*.
  • Content filters: the broker inspects message attributes (region = "EU") and delivers only matches.

Use many specific topics rather than one giant topic with filtering when volumes are high; filtering at the consumer wastes network and CPU.

Python Wildcard routing of topic names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Brokers compared

Traditional broker (RabbitMQ, SQS/SNS, ActiveMQ) Log-based broker (Kafka, Pulsar, Kinesis)
Model Smart broker, tracks each message Append-only log, consumers track offsets
After delivery Message deleted once acked Message kept for a retention period
Replay Not possible Rewind the offset and re-read
Ordering Per queue Per partition
Throughput Thousands to hundreds of thousands per sec Millions per sec
Best for Task distribution, flexible routing Event streams, analytics, event sourcing

In a log-based broker, the topic is split into partitions; the message key decides the partition, which guarantees order per key and lets consumers in a group each own some partitions. That is sharding applied to a message log.

Delivery issues to plan for

  • Duplicates: at-least-once delivery means subscribers must be idempotent, as in Message Queues.
  • Slow subscribers: one slow group should not block others; each group has its own backlog.
  • Schema evolution: publishers and subscribers deploy independently. Add fields rather than renaming, version event types, and tolerate unknown fields.
  • Poison messages: use retry limits and a dead letter queue per subscription.
  • Ordering across topics: none is guaranteed, so do not assume PaymentCaptured arrives after OrderPlaced from a different topic unless your design enforces it.

Pub/sub also appears at smaller scale: in-process event emitters, Redis Pub/Sub (fire-and-forget, no history) and browser WebSockets for live updates. Know whether your tool stores messages or only delivers to whoever is connected right now.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Three services subscribe to the same topic, each as its own group. A message is published. How many services receive it?

    Pub/sub fans out one copy to every subscribing group. Inside each group, one instance handles it.
  2. 2. What does a consumer group provide?

    Groups split a topic's messages between their instances, so a service scales out without receiving duplicates from the same group.
  3. 3. What can a log-based broker do that a traditional broker usually cannot?

    Log-based brokers retain messages for a retention period, so consumers can reread history.
  4. 4. In Kafka, what guarantees ordering for events about the same order ID?

    Order is guaranteed within a partition, and the key decides the partition.

Summary

  • Pub/sub delivers one event to many independent subscribers; queues deliver each message to one worker.
  • Consumer groups combine both: every service sees every event, its instances share the work.
  • The broker decouples publishers and subscribers in time, space and speed.
  • Traditional brokers delete after ack; log-based brokers retain and allow replay, with order per partition.
  • Plan for duplicates, schema changes, slow subscribers and poison messages.

Next: Event-Driven Architecture