Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Communication

Queues, Pub/Sub and Event Logs

Compare point-to-point queues, publish/subscribe and replayable logs. Learn ordering, offsets and consumer groups by building a mini log broker.

Direct calls couple two services tightly: the caller must know who to call, and the callee must be up right now. A message broker sits in the middle and removes both limits. Producers hand messages to the broker; consumers pick them up when they are ready. The producer doesn't care who reads them or when.

There are three broker shapes you will meet again and again: the queue (each message goes to one worker), publish/subscribe (each message goes to every subscriber) and the log (an append-only history that readers can replay). Modern systems such as Kafka are built around the log, so we will build a tiny one in Python.

Queue vs publish/subscribe

A queue (point-to-point) shares work. Ten messages and three workers means each message is processed by exactly one worker. Use it for jobs: resize this image, send this email.

Publish/subscribe (pub/sub) broadcasts facts. A publisher announces "order 17 was placed" to a topic; every subscriber gets its own copy. The billing service, the email service and the analytics service all react independently, and the publisher doesn't know they exist.

Queue or topic?

Queue Each message goes to ONE worker. Shares work.
Pub/sub Each message goes to EVERY subscriber. Shares facts.

Same broker idea, different delivery rule.

1 / 2
Python Queue vs pub/sub

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The problem with deleting messages

In a classic queue the broker removes a message once a consumer takes it. That is simple, but it has costs: you can't re-read it if your code had a bug, a second consumer can't get the same data, and "what happened last Tuesday?" has no answer.

An event log keeps messages. It is an append-only sequence where each message has a number, its offset: 0, 1, 2, and so on. Reading doesn't delete anything. Instead, each reader keeps its own committed offset: "I have processed everything before offset 5."

flowchart LR
    P["Producer"] --> L["Log: 0 | 1 | 2 | 3 | 4 | 5"]
    L --> A["Reader A<br/>offset 6"]
    L --> B["Reader B<br/>offset 3"]

One log, two readers at different positions. Neither affects the other.

Partitions and ordering

One log on one machine can't scale forever, so a topic is split into partitions, each its own log. A producer picks the partition by hashing a key (such as the user ID), so all messages with the same key land in the same partition.

This gives the key ordering rule: order is guaranteed only inside a partition. All events for user 42 arrive in order because they share a partition, but events for user 42 and user 43 in different partitions can be read in any relative order. If your logic needs a global order, you pay in throughput; most systems choose "ordered per key" and design around it.

Consumer groups

A consumer group is a set of workers that share one topic's work. The rule: each partition is read by exactly one member of the group at a time. Add members to scale out (up to the number of partitions), and if a member dies its partitions move to the others. Different groups are independent, each with its own offsets, which is how a log gives you both queue behavior (workers in one group) and pub/sub behavior (several groups).

A mini log broker

Below is a small broker with partitions, key-based routing, consumer groups with committed offsets, and replay. It uses zlib.crc32 as the hash so the key-to-partition mapping is the same on every run (Python's built-in hash() of a string changes between runs).

Python A mini log broker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that all of user-1's events (login, add-to-cart, checkout) share one partition and keep their order, and that the new group analytics sees the whole history even though billing already read it.

Offsets, crashes and delivery guarantees

Where a consumer commits its offset decides its delivery guarantee, as discussed in Messages, RPC and Delivery Guarantees:

  • Process, then commit gives at-least-once. If the consumer crashes after processing but before committing, it restarts from the old offset and processes some messages again.
  • Commit, then process gives at-most-once. A crash after the commit loses the unprocessed messages.
  • With processing that is idempotent (see Idempotency, Retries and Backoff), at-least-once behaves like exactly-once.
Python Crash and replay with committed offsets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The consumer crashed after 6 messages but had committed only up to offset 4, so after the restart two messages were processed twice. Nothing was lost, which is the at-least-once trade. Committing more often shrinks the duplicate window but adds commit overhead.

Choosing between them

Need Pick
Background jobs shared by workers, deleted when done Queue (SQS, RabbitMQ work queues)
Broadcast notifications to several independent services Pub/sub (SNS, Redis pub/sub, topics)
Ordered history, replay, many independent readers, stream processing Log (Kafka, Pulsar, Redpanda)

Also remember retention: a log keeps data for a configured time or size, so replay works only within that window. And backpressure: if consumers are slower than producers, the backlog (called lag: the producer's latest offset minus the consumer's committed offset) grows. Watching consumer lag is the first monitoring signal for any broker. Read more in resilience patterns.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a point-to-point queue with three workers, how many workers process a given message?

    A queue shares work: each message is delivered to a single consumer. Pub/sub is the style that delivers a copy to every subscriber.
  2. 2. Where does a log broker guarantee message order?

    Each partition is its own ordered log. Hashing by key puts related messages in one partition, so they stay in order; messages in different partitions have no guaranteed relative order.
  3. 3. A consumer processes a message and crashes before committing its offset. What happens after restart?

    The committed offset is still before that message, so the consumer re-reads and re-processes it. Idempotent processing makes that harmless.
  4. 4. What is consumer lag?

    Lag measures how far behind a consumer is. A growing lag means consumers cannot keep up with producers.

Summary

  • A broker decouples producers from consumers in time and identity.
  • A queue gives each message to one worker; pub/sub gives each subscriber a copy.
  • A log is append-only: messages have offsets, reading doesn't delete, and each reader tracks its own committed offset, so replay is possible.
  • Partitions scale a topic; ordering is guaranteed per partition, and keys route related messages together.
  • A consumer group shares partitions among workers; separate groups read independently.
  • Commit after processing for at-least-once and make handlers idempotent; watch consumer lag.

Next: Replication: Leaders and Followers