Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 42 ยท Projects

Project: A Reliable Job Queue

Build a job queue on Redis with producers and workers, delayed jobs, retries with exponential backoff, a visibility timeout and a dead-letter list.

Almost every web application eventually needs to do work later: send an email, resize an image, call a slow partner API. The user should not wait for it, so the request just writes a job into a queue and a pool of workers picks the jobs up. The simplest Redis queue is a list: LPUSH to add, BRPOP to take. It works until the first worker crashes in the middle of a job, because the job was already removed from the list and nobody remembers it existed.

A reliable queue never loses a job it has accepted. This project builds one step by step with four ingredients: a visibility timeout (a claimed job comes back if the worker disappears), retries with exponential backoff (failed jobs wait a little longer each time), delayed jobs (run this at a given time) and a dead-letter list (jobs that keep failing are set aside for a human instead of looping forever).

Real Redis cannot run on this page. The code runs against a plain JavaScript emulator with the real semantics of the list, hash and sorted-set commands used here, and a fake clock that we move by hand, so timelines are identical on every run. The claim function below would be a Lua script in real Redis (see Lua scripting); the emulator runs it as one uninterrupted JavaScript function, which gives it the same all-or-nothing behaviour that a script has.

Four places a job can live

ready A list of job ids waiting for a worker.
delayed A sorted set scored by run time.
active A sorted set scored by deadline.
dead A list of jobs that gave up.
1 / 2

The data model

Every job is a small hash job:ID with its type, JSON payload, the number of attempts so far and the maximum allowed. Queues hold only job ids, never the payload, so moving a job between states is cheap. A counter jobs:next-id hands out ids.

Four structures describe where a job currently is:

  • jobs:ready is a list. Producers LPUSH, workers RPOP, so jobs leave in the order they arrived.
  • jobs:delayed is a sorted set whose score is the time (in milliseconds) when the job may run. A sorted set is the natural "timer wheel" in Redis: ZRANGE ... BYSCORE -inf <now> returns exactly the jobs whose time has come.
  • jobs:active is a sorted set whose score is the deadline, the moment after which the job is considered abandoned. A claimed job is in this set and in no other.
  • jobs:dead is a list of ids that exhausted their attempts. Their hashes stay, with the last error recorded.
stateDiagram-v2
  [*] --> ready: enqueue
  [*] --> delayed: enqueue with delay
  delayed --> ready: time has come
  ready --> active: claim
  active --> [*]: ack
  active --> delayed: fail or timeout, with backoff
  active --> dead: attempts used up

A job is in exactly one place at a time; each arrow is one small Redis operation.

Claiming, acknowledging and the visibility timeout

claim does four things in one go. It promotes delayed jobs that are due, reaps active jobs whose deadline passed, pops one id from jobs:ready and registers it in jobs:active with a deadline. The worker then runs the job and calls ack, which removes the id from jobs:active and deletes the hash.

Look at how every state change is guarded by the return value of ZREM. ZREM returns 1 only for the caller that really removed the member. If two workers both notice that a job's deadline has passed, only one gets 1, and only that one puts the job back. The same trick protects ack: if a slow worker finishes after its deadline, the job was already taken away and possibly handed to someone else, so ZREM returns 0 and ack reports failure. The late worker's result is simply ignored, which is why handlers must be idempotent: a job can run twice, once by the slow worker and once by its replacement. Redis queues give you at-least-once delivery; exactly-once is something your handler (for example a unique key per payment) has to provide.

The first TryIt follows three jobs through this machinery. Watch the depth() line, which counts each of the four places.

JS Producers, workers and a visibility timeout

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the middle of the output carefully. After the clock jumps 2.5 seconds, w2 asks for work and gets nothing, yet delayed has grown from 1 to 2: the reaper found job 2 past its deadline and moved it to the delayed set to wait out a one-second backoff. When w1 finally calls ack, it gets false. One second later both delayed jobs are due, w2 receives job 3 and then job 2, and job 2 shows attempt 2: the abandoned run counted as an attempt.

Retries, backoff and dead letters

A failing job can fail for two kinds of reasons. A transient failure (a timeout, a rate limit) will probably succeed later. A permanent failure (a payload that can never be parsed) never will. A queue cannot always tell them apart, so it treats every failure the same way: try again after a delay, a limited number of times, then give up loudly.

retryOrBury implements that policy. It reads attempts and maxAttempts from the hash. While attempts remain, the job goes back to jobs:delayed with a delay of BACKOFF_MS * 2 ** (attempts - 1): one second, two seconds, four seconds. The growth gives a struggling downstream service room to recover, instead of hammering it at full speed. Real systems usually add jitter, a random fraction of the delay, so that a thousand jobs that failed together do not all come back at the same millisecond. We leave it out so the output is predictable. When the attempts are used up, the id goes to jobs:dead with the last error stored on the job hash, where an operator can inspect it, fix the bug and re-queue the job.

The second TryIt creates five kinds of work and two workers. One job succeeds at once, one needs three attempts (report), one never succeeds (poison), one makes its worker crash without acknowledging (crash), and one is delayed. Each simulated second, each live worker claims at most one job.

JS Retries, backoff, a crashed worker and the dead-letter list

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Trace the story in the log. The report job fails twice and succeeds on attempt 3, with gaps of 1 and 2 seconds between its attempts. The poison job gets three attempts and then lands in the dead-letter list with its error. The crash job is the interesting one: the worker that took it never reports back, and nothing happens for five seconds. At t+7s the surviving worker's claim reaps it, and after a one-second backoff it runs again at t+8s and succeeds. Without the visibility timeout that job would have been lost, which is exactly what a bare RPOP queue does.

Production notes

  • Atomic claim. A bare RPOP followed by ZADD has a crash window between the two commands. Put them in a Lua script, or use LMOVE into a per-worker processing list and a sweeper, as described in queues, locks and leaderboards.
  • Blocking waits. Real workers do not poll in a loop. They wait with BLMOVE or BRPOP, or sleep until the earliest delayed job is due.
  • Heartbeats. A job that legitimately takes longer than its timeout should extend its own deadline with another ZADD jobs:active <new deadline> <id> while it runs.
  • Persistence. A queue holds work you cannot recreate. Turn on the append-only file (persistence) and remember that a Redis restart without it empties every queue.
  • Streams as an alternative. Streams with consumer groups give you pending lists and XAUTOCLAIM out of the box, but no delayed jobs, so many systems keep a sorted set for timers and a stream for ready work.
  • Existing libraries. Queue libraries for most languages implement variations of this design. Learn the model here, then use a maintained library in production.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does the visibility timeout solve?

    A claimed job gets a deadline. If it is not acknowledged before then, the reaper treats the worker as lost and the job is retried.
  2. 2. Why is the result of ZREM checked before putting a job back or deleting it?

    ZREM returns 1 for the caller that removed the member and 0 for everybody else, which turns it into a tiny race-free ownership check.
  3. 3. A worker finishes a job after its deadline. What does ack do in this design?

    The job left jobs:active when the deadline passed, so ZREM returns 0. The late result is ignored, and the replacement worker's run is the one that counts.
  4. 4. Why do retries use exponential backoff, ideally with jitter?

    Growing delays reduce pressure on a struggling service, and random jitter spreads out jobs that failed at the same moment.
  5. 5. What delivery guarantee does this queue provide?

    A job is retried until it is acknowledged or exhausted, so after a timeout it can run twice. Exactly-once effects must be built into the handler.

Summary

  • Keep job data in a hash and queue only ids in four places: a ready list, a delayed sorted set, an active sorted set and a dead-letter list.
  • Claimed jobs get a deadline in jobs:active; a reaper returns the ones that passed it, so a crashed worker never loses a job.
  • Guard every transition with the ZREM result so exactly one actor owns each move, and make claim atomic with Lua in production.
  • Retry with exponential backoff (and jitter), count attempts, and park jobs that exhaust them in a dead-letter list with their last error.
  • The queue is at-least-once: write idempotent handlers.

Next project: Cached API with Rate Limiting.