Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท Reliability and Acknowledgements

Durability and Persistent Messages

Learn what survives a RabbitMQ restart: durable exchanges and queues, delivery_mode 2 persistent messages, transient queues, lazy behavior and the full checklist for no message loss.

"Will my messages survive a restart?" is the first reliability question every RabbitMQ user asks, and the answer is "it depends on three separate switches". RabbitMQ keeps two kinds of things: entities (exchanges, queues, bindings) and messages. Each can be durable or transient, and a message only survives if both its queue and the message itself are marked to survive.

This lesson explains each switch, what a crash really does, and the checklist that combines durability with publisher confirms and consumer acknowledgements. The real broker cannot run on this page: pika and rabbitmqctl snippets are read-only, and the runnable blocks are a plain Python stand-in that simulates a broker restart. It is a model of the rules, not of RabbitMQ's storage engine.

The three switches

What survives a restart

Durable exchange Definition is stored.
Durable queue Definition is stored.
Persistent message Body is written to disk.
All three Message comes back after restart.

Miss one and the message can vanish.

1 / 2
  1. Durable exchange (durable=True on exchange_declare). Without it the exchange disappears on restart and publishers get NOT_FOUND errors until it is redeclared.
  2. Durable queue (durable=True on queue_declare). The queue definition is stored. A durable queue with transient messages still comes back, but empty.
  3. Persistent message (delivery_mode=2 in the properties, or delivery_mode=Persistent in other clients). The broker writes the body to disk. Persistent messages published to a transient queue are pointless: the queue is deleted at restart.
Python
channel.exchange_declare(exchange="shop", exchange_type="topic", durable=True)
channel.queue_declare(queue="orders", durable=True)
channel.queue_bind(queue="orders", exchange="shop", routing_key="order.*")

channel.basic_publish(
    exchange="shop",
    routing_key="order.created",
    body=b'{"id": 7}',
    properties=pika.BasicProperties(
        delivery_mode=pika.DeliveryMode.Persistent,
        content_type="application/json",
    ),
)

Bindings between two durable entities are also durable. A binding to a transient queue disappears with it.

Quorum queues and streams are always durable. Declaring them with durable=False is rejected, and the delivery_mode property is effectively ignored because every message is replicated and written to disk. Classic queues are the ones where you must choose. See Quorum Queues.

What happens at a crash

RabbitMQ does not fsync every message individually. Persistent messages are written to the queue's journal and flushed in batches. This keeps throughput high, but it means there is a small window between "the broker received the message" and "the message is on disk". If the machine loses power in that window, the message is lost even though it was marked persistent.

The way to close that window is publisher confirms: for a persistent message in a durable queue, the broker sends basic.ack only after the message was flushed. The producer then knows the message is safe and can drop its copy. Without confirms, "persistent" is a best effort.

On the consumer side, unacknowledged messages are written too, so a delivered-but-unacked message is redelivered after a restart, with redelivered=true.

Python Stand-in: restart with different settings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only the first combination returns a message. The second returns an empty queue, the other two lose the queue itself.

Queue lifetime options

Durability is separate from queue lifetime:

  • exclusive=True: usable by one connection only and deleted when it closes. Used for private reply queues.
  • auto_delete=True: deleted when its last consumer unsubscribes.
  • x-expires: deleted after the queue was unused for that many milliseconds.

A server-named queue (queue="") is usually exclusive, auto-delete and non-durable: a good fit for temporary subscribers, a bad one for work you must not lose.

Memory, disk and queue length

Persistence is not the same as being "paged out". Classic queues (since the version 2 storage engine in RabbitMQ 3.12, the only engine in 4.x) keep a small in-memory cache and read the rest from disk, so very long queues no longer exhaust memory the way they once did. Still, a backlog of millions of messages is a symptom, not a feature. Limit it:

Text
rabbitmqctl set_policy limits "^orders$" \
  '{"max-length": 100000, "overflow": "reject-publish"}' --apply-to queues

overflow: reject-publish makes the broker nack publishes to a full queue (visible through confirms), while the default drop-head silently discards the oldest message. See Dead Letter Exchanges and Retries to keep rejected messages instead of losing them.

A design that does not lose messages

Here is the complete chain, from producer to consumer, as a checklist you can use in reviews.

Python Stand-in: a no-loss checklist

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even a perfect setup has limits. A single disk failure takes a classic queue with it (replicate with quorum queues), and at-least-once delivery means duplicates are possible, so consumers should handle them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You publish a persistent message (delivery_mode=2) to a non-durable queue and restart the broker. What do you find?

    The queue definition is not stored, so the queue is deleted at restart. Persistent messages need a durable queue to survive.
  2. 2. Which mechanism tells the producer that a persistent message has actually reached disk?

    RabbitMQ flushes persistent messages in batches. With confirms enabled it sends basic.ack only after that flush.
  3. 3. What does a durable queue holding only transient messages contain after a restart?

    The queue definition survives, the transient message bodies do not.
  4. 4. How do quorum queues differ regarding durability?

    Quorum queues and streams are always durable; every message is replicated and written to disk.

Summary

  • A message survives a restart only if its exchange and queue are durable and the message is persistent (delivery_mode=2).
  • Persistent messages are flushed in batches; combine them with publisher confirms and manual consumer acks.
  • Quorum queues and streams are always durable and replicated; exclusive and auto-delete queues are temporary by design.
  • Cap backlog with max-length policies and decide the overflow behavior on purpose.

Next: Dead Letter Exchanges and Retries