Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Reliability and Acknowledgements

Dead Letter Exchanges, TTL and Retries

Learn RabbitMQ dead lettering: x-dead-letter-exchange, the x-death header, message TTL, delivery limits and the delayed retry queue pattern, with a runnable simulation.

Some messages cannot be processed: the payload is malformed, a dependency is down, or a bug throws every time. If you nack with requeue=true the message comes straight back and spins in a loop. If you nack with requeue=false it vanishes. Neither is acceptable for orders or payments. RabbitMQ's answer is the dead letter exchange (DLX): a message that is "dead" is republished to an exchange you choose, where it can be stored for inspection, retried later or alerted on.

This lesson covers when messages die, how to configure a DLX, the x-death header, TTLs, and the classic delayed retry pattern built from them. Real RabbitMQ cannot run on this page: the pika and policy snippets are read-only, and the runnable blocks are a plain Python stand-in for the routing rules. It models behavior; it does not use the real broker.

When does a message become dead?

A message is dead-lettered for exactly these reasons (the reason is recorded as x-first-death-reason and in x-death):

Reason Trigger
rejected A consumer called basic.nack or basic.reject with requeue=false.
expired The message TTL ran out.
maxlen The queue exceeded x-max-length (or x-max-length-bytes) with overflow: drop-head.
delivery_limit A quorum queue message was returned more times than x-delivery-limit.

A message is not dead-lettered when you simply ack it, when a queue is deleted, or when overflow is reject-publish.

flowchart LR
    P[Producer] --> X((orders))
    X --> Q[orders queue]
    Q -->|nack requeue=false<br/>TTL expired<br/>queue full| DLX((orders.dlx))
    DLX --> DLQ[orders.dead queue]
    DLQ --> Ops[Alert or manual replay]

Configuring a DLX

Set the dead letter exchange on the queue with arguments, or better with a policy, which you can change without redeclaring the queue:

Python
channel.exchange_declare(exchange="orders.dlx", exchange_type="fanout", durable=True)
channel.queue_declare(queue="orders.dead", durable=True)
channel.queue_bind(queue="orders.dead", exchange="orders.dlx")

channel.queue_declare(
    queue="orders",
    durable=True,
    arguments={
        "x-dead-letter-exchange": "orders.dlx",
        # optional: "x-dead-letter-routing-key": "orders.failed",
    },
)
Text
rabbitmqctl set_policy dlx "^orders$" \
  '{"dead-letter-exchange":"orders.dlx"}' --apply-to queues

If x-dead-letter-routing-key is not given, the message keeps its original routing key. Dead lettering reuses the normal routing rules, so the DLX must have a binding that matches or the message is dropped, and nothing warns you. Test your DLX path before production.

When a message is dead-lettered, RabbitMQ adds an x-death header: an array with one entry per (queue, reason), holding queue, reason, count, exchange, routing-keys and time. Because it counts how often the message died in each queue, x-death[0].count is a handy retry counter.

Python Stand-in: nack to a dead letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Message TTL and queue TTL

TTL (time to live) makes messages expire:

  • Per-queue TTL: x-message-ttl (policy key message-ttl) in milliseconds. Every message in the queue gets that lifetime.
  • Per-message TTL: the expiration property (a string of milliseconds, e.g. "60000").
  • If both are set, the smaller value wins.
  • x-expires is different: it deletes the queue after a period without use.

Expired messages become dead letters with reason expired. A subtle rule: in classic queues, expiry is only checked at the head of the queue. A message with a short TTL stuck behind a message with a long TTL is not removed until it reaches the head. Per-queue TTL does not have this issue because all messages share one TTL and order equals expiry order. So prefer per-queue TTL for retry delays.

The delayed retry pattern

RabbitMQ has no built-in "retry in 30 seconds". You build it with a TTL queue that dead-letters back to the original exchange:

flowchart LR
    X((work)) --> W[work queue]
    W -->|nack requeue=false| R((retry))
    R --> RQ["work.retry<br/>x-message-ttl 30000<br/>DLX = work"]
    RQ -->|expired| X
    W -->|too many failures| D[work.dead]
  1. Consumers nack(requeue=False) a failing message. The work queue's DLX is the retry exchange.
  2. The work.retry queue has no consumers, a message TTL of 30 seconds and its own DLX pointing back to the work exchange.
  3. After 30 seconds the message expires and returns to the work queue. The consumer reads x-death and, when the count passes a limit, publishes the message to a parking-lot queue (work.dead) and acks it instead of failing again.
Python Stand-in: retry loop with a parking lot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quorum queues: delivery limit and at-least-once dead lettering

Quorum queues add two helpers. x-delivery-limit (default 20 in RabbitMQ 4.x) counts how many times a message was returned to the queue; when exceeded, the message is dead-lettered (reason delivery_limit) or dropped if no DLX exists. This stops poison messages without any retry queues. And with x-dead-letter-strategy: at-least-once (on a quorum queue, with overflow: reject-publish) dead lettering itself is made reliable, which is not the case for classic queues, where a dead letter can be lost if the target is unavailable.

Treat dead letter queues like any other queue: monitor their length (see Monitoring and Prometheus), and have a way to replay messages after fixing the bug.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these does NOT cause a message to be dead-lettered?

    A normal ack removes the message. Rejection without requeue, expiry, maxlen and the quorum delivery limit are the dead letter triggers.
  2. 2. A queue has x-dead-letter-exchange set but the DLX has no matching binding. What happens to a dead message?

    Dead lettering uses ordinary routing. If nothing matches, the message is dropped, so test the DLX path.
  3. 3. In the delayed retry pattern, why does the retry queue have no consumers?

    The retry queue is only a timer: expiry triggers dead lettering back into the work queue after the delay.
  4. 4. Why is per-queue TTL preferred over per-message TTL for retry delays in classic queues?

    With one TTL for the whole queue, order equals expiry order, so the head always expires first.

Summary

  • Dead lettering republishes rejected, expired, overflowed or over-delivered messages to a DLX instead of losing them.
  • Configure x-dead-letter-exchange by policy; make sure the DLX has a matching binding; read the x-death header for counts and reasons.
  • Retries are built from a TTL queue with no consumers whose DLX points back to the work exchange; add a parking-lot queue after a maximum number of tries.
  • Quorum queues provide a delivery limit and at-least-once dead lettering.

Next: Clustering Fundamentals