Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Projects

Project: Reliable Job Queue

Build a job queue with visibility timeouts, retries, a dead-letter list and idempotent workers, then prove with a simulation that at-least-once processing is safe.

Almost every real system has a queue somewhere: send the welcome email, resize the photo, charge the card. The producer drops a job into the queue and moves on; workers pick jobs up later. The hard part is not the happy path. It is what happens when a worker crashes halfway, when a job always fails, or when the network loses the "done" message.

In this project you build the core of a reliable queue, in the style of Amazon SQS or RabbitMQ with acknowledgements, and then prove with a simulation that it behaves. You will use ideas from delivery guarantees, idempotency and retries and queues. The whole thing runs on a virtual clock, so every run is deterministic.

Four rules of a reliable queue

Visibility timeout A taken job is hidden, not deleted.
Ack to delete Only a finished worker removes the job.
Retry If no ack arrives in time, the job reappears.
Dead letters After too many tries, park the job for humans.

Together these give at-least-once delivery.

1 / 2

The design

A queue server that never loses a job cannot simply delete a job when a worker takes it. If the worker dies, the job is gone. Instead:

  1. receive() hands out a job and sets invisible_until = now + visibility timeout. Other workers do not see it until then.
  2. When the worker finishes, it calls ack(job_id), which deletes the job for good.
  3. If the timeout passes without an ack, the job becomes visible again and some worker receives it a second time.
  4. Each receive increments an attempts counter. A job that exceeds max_attempts is moved to a dead-letter list so that one poison job cannot loop forever and hog workers.
stateDiagram-v2
    [*] --> Visible: send
    Visible --> InFlight: receive
    InFlight --> Done: ack
    InFlight --> Visible: timeout expires
    Visible --> Dead: attempts exceeded
    Done --> [*]

A job is only removed by an ack. Every other path brings it back or parks it.

Step 1: visibility timeouts

Here is the queue itself. Time is an integer you advance by hand, so there are no real sleeps.

Python A queue where a crashed worker's job comes back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output carefully. Worker C finds nothing at t=0, because both jobs are in flight. Worker B acks j2, so it is gone. Worker A crashed while holding j1, so nobody acks it, and when the timeout expires at t=5 the job comes back with attempts raised to 2. This is at-least-once delivery: the queue prefers delivering a job twice to losing it.

Choosing the visibility timeout is a real trade-off. Too short, and a slow but healthy worker has its job handed to a second worker while it is still running. Too long, and a crashed worker's job sits idle for minutes. Real queues let a worker extend the timeout (a heartbeat) while it is still busy.

Step 2: failures you must expect

The queue can fail a job in three different ways, and each needs a different answer:

Failure What the queue sees Result
Worker crashes before the work No ack Job returns after the timeout, harmless
Worker crashes after the work, before the ack No ack Job returns, the work runs twice
Job is poison (always raises an error) No ack, forever Attempts run out, job goes to the dead-letter list

The second row is the dangerous one. The queue cannot tell "crashed before" from "crashed after", so it must redeliver. The fix lives in the worker, not the queue: make the work idempotent.

Step 3: idempotent workers

An idempotent worker remembers which job IDs it has already applied. If a duplicate arrives, it skips the effect and just acks. In production that memory is a database table with a unique key on the job ID, and the effect and the ID are saved in one transaction; here it is a Python set.

The simulation below processes ten jobs that each add an amount to a balance. Worker crashes are random (seeded) and happen after the effect but before the ack. Job 4 is poison. We run the same scenario twice: with a naive worker and with an idempotent one.

Python Naive worker vs idempotent worker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive worker ends with a balance of 94 instead of 50, because every job whose ack was lost was applied again. The idempotent worker lands exactly on 50. In both runs job4 ends up in the dead-letter list after four attempts instead of cycling forever. The naive run also dead-letters job7: it was applied several times and its ack was lost every time. The number of deliveries (17 to 20) is far higher than the ten jobs: that is the price of at-least-once.

Step 4: test the invariants

A queue is easy to get subtly wrong, so state the promises as invariants and check them over many random schedules, not just one. Ours are:

  • No loss: every job ends either acked or in the dead-letter list, and nothing is left in flight.
  • Exactly one effect: with an idempotent worker, each healthy job's effect is applied exactly once, and a poison job's never.
  • Poison is parked: every job that always fails reaches the dead-letter list.

The check also counts how often a healthy job is dead-lettered anyway. That happens when its ack is lost on every attempt: the effect is applied, but the queue never learns about it. Such a dead letter is a false alarm, and an idempotent worker makes it safe to re-queue.

Python Check the invariants over 200 random seeds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you break the queue on purpose (for example, delete a job inside receive instead of waiting for the ack), the "no loss" check starts failing on seeds with crashes. That is the point of invariant tests: they catch the schedules you did not think of. Testing distributed systems goes deeper on this style.

Where to take it next

  • Ordering: this queue is not FIFO under retries. A retried job can overtake later jobs. If you need order per customer, partition by key, as in sharding.
  • Backoff: instead of a fixed timeout, make the retry delay grow with attempts, with jitter (see backoff).
  • Real brokers: SQS, RabbitMQ and Redis-based queues all follow this ack, timeout and dead-letter shape. Knowing it makes their settings (visibility timeout, max receive count, redrive policy) easy to read.
  • Effect and ack together: the gap between "do the work" and "ack" never fully closes. Idempotency keys, or the outbox idea, shrink the damage.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does receive() hide a job instead of deleting it?

    Deleting on receive would lose the job if the worker died. The job is only removed by an ack.
  2. 2. A worker finishes the work and crashes before sending the ack. What happens?

    The queue cannot tell 'crashed before' from 'crashed after' the work, so it redelivers. That is why workers must be idempotent.
  3. 3. What is the purpose of the dead-letter list?

    A poison job would otherwise be retried forever. After max attempts it is set aside for a human or a repair process.
  4. 4. How does an idempotent worker survive duplicates?

    Checking a stored ID before applying the effect makes a repeat delivery harmless.

Summary

  • A reliable queue hides a received job for a visibility timeout and deletes it only on ack.
  • Missing acks cause redelivery, giving at-least-once delivery and occasional duplicates.
  • A dead-letter list catches jobs that exceed max_attempts.
  • Idempotent workers (a table of applied job IDs, saved atomically with the effect) turn at-least-once into effectively-once.
  • Test with invariants over many seeded schedules: no loss, no double effect, bounded retries.

Next project: Project: Causally Ordered Chat.