Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 65 ยท Projects

Project: A Message Queue with Retries and Dead Letters

Build a message queue in Python: producers, consumers, acknowledgements, visibility timeouts, retry with backoff, a dead-letter queue and idempotent handlers.

Message queues decouple the part of a system that creates work from the part that does it. A web request can enqueue "send welcome email" and return at once; workers handle the email later, retry if the mail server is down, and set aside messages that keep failing. In this project you will build a small queue with the features that make real queues useful: acknowledgements, visibility timeouts, retries with backoff, a dead-letter queue and consumers that tolerate duplicates.

The delivery model

sequenceDiagram
    participant P as Producer
    participant Q as Queue
    participant C as Consumer
    P->>Q: send(message)
    C->>Q: receive()
    Q-->>C: message (hidden for 30 s)
    C->>C: process
    C->>Q: ack(message)
    Note over Q: no ack in time: message reappears

A received message is hidden, not deleted. It is only removed when the consumer acknowledges it.

This gives at-least-once delivery: if a consumer crashes after doing the work but before acknowledging, the message reappears and is processed again. Exactly-once is not something a queue can promise on its own, so consumers must be idempotent (safe to run twice).

Step 1: the queue

Python Queue with visibility timeout and acknowledgements

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: workers, retries and the dead-letter queue

A message that always fails (a poison message, for example malformed data) would loop forever and block progress. After a maximum number of attempts, move it to a dead-letter queue (DLQ) so humans can inspect it. Between attempts, wait using exponential backoff, so a failing dependency gets time to recover.

Python Workers with backoff and a dead-letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The flaky message eventually succeeds once the dependency recovers; the poison message exhausts its attempts and lands in the DLQ without ever blocking a and b.

Step 3: idempotent consumers

Because delivery is at least once, the handler may see the same message twice. Two common techniques: record processed message IDs and skip repeats, or make the operation naturally repeatable (setting a value rather than incrementing it).

Python Duplicate delivery and an idempotent handler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ID check and the update must happen atomically (one database transaction), otherwise a crash between them recreates the problem.

Step 4: extensions

  1. Priorities: two queues, or a priority heap, so urgent messages jump ahead.
  2. Ordering: FIFO per key (all messages for one account go to one partition) rather than global order, which scales.
  3. Pub/sub: deliver each message to many independent consumer groups, each with its own position (like a log).
  4. Backpressure: reject or slow producers when the queue exceeds a size limit.
  5. Metrics: track queue depth, age of the oldest message, retries and DLQ size, and alert on growth.
  6. Replay: keep a message log so a new consumer can start from the past.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the visibility timeout do?

    It lets the queue recover work from crashed consumers.
  2. 2. Why have a dead-letter queue?

    Poison messages are isolated for inspection and manual or automated repair.
  3. 3. At-least-once delivery means consumers must be:

    The same message may arrive more than once, so repeating the work must be safe.
  4. 4. Why use exponential backoff between retries?

    Increasing delays reduce load on a struggling service.

Summary

  • Receiving hides a message; only an acknowledgement removes it, giving at-least-once delivery.
  • Retries with exponential backoff handle temporary failures; a dead-letter queue isolates poison messages.
  • Consumers must be idempotent: track IDs or design repeatable operations, atomically.
  • Watch queue depth and oldest message age to detect trouble early.

Next: Project: A Capacity Planner