Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 37 ยท Clustering and High Availability

RabbitMQ Streams

Learn RabbitMQ streams: append-only replicated logs, non-destructive reads, consumer offsets, retention, super streams and when to choose a stream instead of a queue.

Ordinary queues are destructive: once a consumer acknowledges a message, it is gone. A stream works the other way. It is an append-only log: producers add messages to the end, and consumers read from any position without removing anything. Many consumers can read the same data, a new consumer can start from the beginning, and a crashed consumer can resume or replay from an earlier point. Streams were added in RabbitMQ 3.9 and are replicated like quorum queues.

RabbitMQ cannot run here, so broker snippets are read-only. The runnable blocks are a plain Python stand-in that models a log with offsets and retention. It imitates the idea (offsets, retention, replay), not the storage format or the protocol.

Queue or stream?

Queue versus stream

Queue A message is consumed once, then deleted.
Stream A message stays until retention removes it.
Replay Streams can be read again from any offset.
Fan-out Many readers share one copy.

Pick by how messages are consumed, not by size.

1 / 2

Choose a stream when you need: multiple independent consumers of the same data without one queue per consumer, replay or time travel (debugging, new services catching up), very high throughput (millions of messages per second are reachable with the stream protocol), or large backlogs kept for hours or days. Choose a queue for task distribution where each message is processed once by one worker and then discarded.

Declaring a stream

A stream is declared like a queue with x-queue-type: stream, and must be durable:

Python
channel.queue_declare(
    queue="clickstream",
    durable=True,
    arguments={
        "x-queue-type": "stream",
        "x-max-length-bytes": 20_000_000_000,   # keep about 20 GB
        "x-max-age": "7D",                      # or keep 7 days
        "x-stream-max-segment-size-bytes": 100_000_000,
    },
)

Retention is the only way data leaves a stream: whole segment files are deleted when the stream exceeds x-max-length-bytes or the segment is older than x-max-age (units Y, M, D, h, m, s). Retention works per segment, so the stream can briefly hold slightly more than the limit.

Streams speak two protocols. Normal AMQP 0-9-1 clients can publish and consume (with a prefetch and manual acks), and the dedicated stream protocol on port 5552 (Java, Go, .NET, Python and Rust clients) is much faster and supports offset tracking and deduplication.

Offsets

Every message has a numeric offset: 0 for the first, 1 for the next, and so on. A consumer chooses where to start with x-stream-offset:

Value Meaning
first Start at the oldest retained message
last Start at the last chunk written
next Only new messages (default for AMQP consumers)
a number A specific offset
a timestamp The first message at or after that time
Python
channel.basic_qos(prefetch_count=100)     # required for stream consumers
channel.basic_consume(
    queue="clickstream",
    on_message_callback=handle,
    arguments={"x-stream-offset": "first"},
)

In a stream, basic_ack just returns credit so the broker may send more; it does not delete anything. To resume after a restart, the consumer must remember its last processed offset. With the stream protocol you call store_offset and the broker keeps it for you under a consumer name; with AMQP clients you store it yourself (database, file) or use the x-stream-offset header of the last message.

Python Stand-in: append-only log with offsets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Retention and offset tracking together

The next block simulates a consumer that stores its offset, crashes and resumes, then a second consumer that replays from the start.

Python Stand-in: resume and replay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice offset 4 is processed twice. Streams give at-least-once reading; make the handler idempotent or store the offset in the same database transaction as the result.

Deduplication and publishing

With the stream protocol, a producer gives itself a name and a monotonically increasing publishing id per message. The broker remembers the highest id per producer and ignores repeats, so retries after a lost confirm do not create duplicates. Publisher confirms exist for streams exactly as for queues.

Super streams

One stream is one log on one leader, so it is limited by one node. A super stream is a logical stream split into partitions (each a normal stream), routed by a hash of a key. Producers choose the partition by key (all events for customer 17 land in the same one, preserving order), and consumers use single active consumer per partition to scale. Like Kafka topics, order is guaranteed only inside a partition.

Limits to remember

  • Streams are not a replacement for a database: they are for event data with bounded retention.
  • There are no per-message acknowledgements, TTL, priorities, redelivery or dead lettering. A failing message cannot be requeued; you handle it in code.
  • A stream still keeps its replicas on several nodes (default 3), so storage is multiplied by the replica count.
  • Choose segment size and retention deliberately; consumers slower than retention lose data silently by being moved forward.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does basic_ack do when consuming from a stream?

    Stream consumption is non-destructive. Data leaves only through retention.
  2. 2. How can a new consumer read all retained messages of a stream?

    Consumers pick their starting point: first, last, next, a numeric offset or a timestamp.
  3. 3. Which feature gives a stream higher throughput than a single stream by splitting it into partitions?

    A super stream is a set of partition streams selected by a routing key hash.
  4. 4. Which scenario suits a queue rather than a stream?

    Work distribution where each message is consumed once and removed fits a queue.

Summary

  • A stream is a replicated, append-only log; reading does not delete, retention does.
  • Consumers choose a start offset (first, last, next, number, timestamp) and must remember their position.
  • The stream protocol (port 5552) adds offset tracking, deduplication and high throughput; super streams add partitions.
  • Streams lack per-message ack, requeue and dead lettering, so use them for event data, not tasks.

Next: Network Partitions and Client Failover