Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Task Queue with Retries and Idempotency

Build a reliable background-job system on RabbitMQ: durable work queue, exponential backoff retry queues, a parking lot, idempotency keys and graceful shutdown, with a runnable simulation.

Almost every web application ends up with background jobs: sending emails, generating PDFs, calling slow APIs. In this project you design a reliable job queue on RabbitMQ that handles the three hard parts: retrying failed jobs with growing delays, never running a job's side effects twice, and shutting down without losing work. The pieces come from earlier lessons: work queues, acks and prefetch, dead lettering and confirms.

RabbitMQ cannot run on this page, so the broker code is read-only and the runnable blocks are plain Python stand-ins with a simulated clock. They reproduce the decisions, not the broker.

Design

flowchart LR
    App[Application] -->|job| J((jobs))
    J --> Q[jobs.work]
    Q --> W[Workers]
    W -->|fail: attempt 1| R1[jobs.retry.10s]
    W -->|fail: attempt 2| R2[jobs.retry.60s]
    W -->|fail: attempt 3| R3[jobs.retry.300s]
    R1 -->|TTL expired| J
    R2 -->|TTL expired| J
    R3 -->|TTL expired| J
    W -->|attempts exhausted| P[jobs.parked]
  • One work queue (jobs.work, quorum, durable), consumed with manual acks and a small prefetch.
  • Several retry queues, one per delay (10 s, 60 s, 300 s), each with x-message-ttl and no consumers, whose dead letter exchange points back to jobs. This gives exponential-style backoff with plain per-queue TTLs, which avoids the head-of-line expiry problem of per-message TTLs.
  • A parking lot (jobs.parked) for jobs that fail every attempt. Humans inspect it and replay.
  • A header x-attempt carries the attempt number. The worker, not the broker, decides which retry queue gets the job.
Python
RETRY_QUEUES = [("jobs.retry.10s", 10_000), ("jobs.retry.60s", 60_000), ("jobs.retry.300s", 300_000)]

channel.exchange_declare("jobs", "direct", durable=True)
channel.queue_declare("jobs.work", durable=True, arguments={"x-queue-type": "quorum"})
channel.queue_bind("jobs.work", "jobs", routing_key="work")
channel.queue_declare("jobs.parked", durable=True, arguments={"x-queue-type": "quorum"})

for name, ttl in RETRY_QUEUES:
    channel.queue_declare(name, durable=True, arguments={
        "x-queue-type": "quorum",
        "x-message-ttl": ttl,
        "x-dead-letter-exchange": "jobs",
        "x-dead-letter-routing-key": "work",       # back into the work queue
    })

def on_job(ch, method, props, body):
    attempt = (props.headers or {}).get("x-attempt", 1)
    try:
        run_job(json.loads(body))
        ch.basic_ack(method.delivery_tag)
    except Exception:
        # republish to a retry queue (or parking lot), then ack the original
        target = RETRY_QUEUES[attempt - 1][0] if attempt <= len(RETRY_QUEUES) else "jobs.parked"
        ch.basic_publish("", target, body, pika.BasicProperties(
            delivery_mode=2, message_id=props.message_id,
            headers={"x-attempt": attempt + 1}))
        ch.basic_ack(method.delivery_tag)

The worker publishes first, then acks. If it crashes between the two, the original is redelivered and the retry is duplicated, a safe failure (duplicate) rather than a lost job. That is why the next section matters.

Simulating retries and backoff

Python Stand-in: retry queues with a simulated clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

email-2 succeeds on its third try after waiting 10 s and 60 s; report-9 burns through all attempts and is parked with its history.

Idempotency: making duplicates harmless

Retries, redeliveries after crashes and publisher retries all mean a job can run more than once. The side effect, such as charging a card or sending an email, must happen once. Three common techniques:

  1. Natural idempotence. "Set status to shipped" can run twice safely. Prefer operations like that.
  2. Idempotency key with a unique constraint. Give each job a message_id (a UUID chosen when the job is created, stored with it). Before doing the work, insert the key into a processed_jobs table with a unique index; if the insert fails, skip. Better: do the insert in the same database transaction as the business change so a crash cannot leave one without the other.
  3. Pass the key to the other system. Many payment and email APIs accept an Idempotency-Key header and deduplicate for you.
Python Stand-in: idempotency key with a unique constraint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unique key and the side effect share a transaction, so either both happen or neither does. When the side effect is an external call (an email API), the ordering matters: record "started", call with an idempotency key, then record "finished"; a crashed job is retried and the external API recognizes the key.

Graceful shutdown and timeouts

  • On SIGTERM, stop consuming first (basic_cancel), let the current job finish, ack it, then close the channel. Messages in prefetch that were not started return to the queue when the channel closes.
  • Make job time bounded: apply your own timeout per job. A job that runs past the broker's consumer_timeout (default 30 minutes) kills the channel and requeues everything.
  • Use prefetch 1 for long jobs, and a separate connection for publishing retries (see Flow Control and Resource Alarms).
  • Use heartbeats and run long CPU work in a thread or subprocess so the connection loop keeps answering heartbeats.
Python Stand-in: graceful shutdown keeps unstarted work

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Production checklist

  • Persistent messages, quorum queues, confirms on publish.
  • Retry queues with TTL, a parking lot and an alert on its depth (Monitoring and Prometheus).
  • message_id on every job and an idempotency key at every side effect.
  • Per-job timeouts, graceful shutdown, small prefetch.
  • Metrics: queue depth, age of the oldest job, attempts per job, parked count.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use separate retry queues per delay instead of per-message TTL in one queue?

    With one TTL per queue, order equals expiry order, so messages return exactly when their delay is over.
  2. 2. In the failure handler, why publish to the retry queue before acking the original?

    Acking first could lose the job if the worker crashes before republishing; duplicates are handled by idempotency.
  3. 3. What makes an idempotency key effective?

    The database enforces that the job id is recorded only once, atomically with the business change.
  4. 4. A worker receives SIGTERM. What is the right order?

    Stop taking new work, complete and ack what is running; unstarted deliveries are requeued when the channel closes.

Summary

  • Build retries from per-delay TTL queues that dead-letter back to the work exchange, with a parking lot at the end.
  • Publish the retry before acking the failed delivery; accept duplicates and neutralize them with idempotency keys.
  • Record the key in the same transaction as the side effect, or pass it to the external API.
  • Bound job time, shut down gracefully and monitor depth, age and parked jobs.

Next: Practice: Troubleshooting Stuck Queues