Skip to content
Tutorials Center
My learning

Topics

Lesson 119 of 133 ยท Projects

Project: Background Job Queue

Build a persistent job queue in Node.js with node:sqlite and worker threads. Atomic claims, retries with backoff, dead jobs, crash recovery with leases and graceful shutdown.

Some work does not belong inside a web request. Sending an email, resizing an image, building a report or calling a slow partner API can take seconds, can fail, and should not make a visitor wait. The standard answer is a background job queue: the request writes a small description of the work ("send welcome email to user 42") and returns at once, and a separate worker picks the job up and does it later.

In this project you build such a queue from scratch. Jobs are stored in SQLite so they survive restarts, handlers run in worker_threads so heavy work never blocks the main thread, failed jobs are retried with exponential backoff, and jobs that keep failing end up in a "dead" state where you can look at them. You will also handle the two hard parts of any queue: two workers must never run the same job, and a worker that crashes in the middle of a job must not lose it. Everything runs in the editor.

Plan: four small pieces

  1. The store (SQLite): the only place that knows about job state. It offers enqueue, claim, complete, fail and recoverStale.
  2. The handlers: plain functions, one per job type, that take a payload and return a result or throw.
  3. The worker pool: a few worker threads. The pool claims jobs from the store, hands them to idle workers, and records the outcome.
  4. The lifecycle: starting, draining, stopping gracefully and recovering after a crash.

A job moves through four states:

State Meaning
pending Waiting for its run_at time and for a free worker
running Claimed by a worker, with a lock that expires (a "lease")
done Finished; the result is stored
dead Failed too many times (or failed permanently); the last error is stored

A failed job that still has attempts left goes back to pending with a later run_at. That one rule gives you retries and backoff without any timers in the database.

The editor runs your code with Node.js 24, where node:sqlite is built in. In that version the module is still marked experimental in the documentation, so its API may change in later releases. It is perfect for a project like this one. Production systems more often use a server database such as PostgreSQL, or Redis-based queues, but the ideas below transfer one to one. See SQLite in Node.js and worker threads if you want a refresher.

Stage 1: claiming a job without a race

The most important method of a queue is the one that hands out work. The obvious way to write it is two steps: SELECT the oldest pending job, then UPDATE it to running. That is a bug waiting for load. Between the two steps another worker can run the same SELECT and see the same job, and now two workers do the same work: two emails, two charges.

You cannot see this bug with one thread, since each call finishes before the next starts, so the example below interleaves the steps by hand: worker B looks before worker A has written anything.

Node.js Why claiming needs one statement

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The unsafe version prints worker A runs job 1, worker B runs job 1: the same job twice. The safe version moves the choice into the UPDATE, as WHERE id = (SELECT ...), and asks for the changed row back with RETURNING. SQLite runs a single statement atomically, so one claim sees the effects of the previous one. A gets 1, B gets 2, and C gets undefined because nothing is left. (If you ever must use two statements, make the second one a compare-and-set, UPDATE ... WHERE id = ? AND status = 'pending', and check changes.)

The race above is not specific to SQLite. With PostgreSQL the same idea is SELECT ... FOR UPDATE SKIP LOCKED, and with Redis it is an atomic BLMOVE or a Lua script. Whatever the storage is, the claim must be one atomic step.

Stage 1 (continued): the full store

Now the whole store. Read it top to bottom; it is the entire "database layer" of the queue.

  • enqueue inserts a row and returns its id. delayMs schedules the job in the future (run_at), and maxAttempts bounds the retries.
  • claim uses the atomic statement from above, also increments attempts, and sets locked_until, the lease. Payloads are stored as JSON text.
  • fail decides between a retry and the dead state. The delay before the next attempt is backoffMs * 2 ** (attempts - 1): 1 s, 2 s, 4 s, ... Backing off gives a struggling service time to recover, and avoids a storm of immediate retries. Passing { retry: false } kills the job at once, for errors that retrying cannot fix.
  • recoverStale puts running jobs whose lease has expired back to pending. It is the safety net for crashed workers; more on it in stage 3.

The store takes a now function as an option. Production code uses Date.now, tests pass a variable they control, so a "one second later" check does not need a real second of waiting. Injecting the clock is the single best trick for testing anything time-based (see mocking and timers).

Node.js Stage 1: the job store and its tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Follow the first part: three jobs are queued, one claimed (running), then job 2 fails once and goes back to pending with a wait of 1000 ms. Claiming right away returns null, since it is not due. After the clock moves 1000 ms, it is claimable again as attempt 2. It fails again, and because max_attempts for that job was 2 the store answers dead. The delayed job only appears after another 5000 ms.

The four checks pin the behavior that matters most: oldest first, exactly doubling backoff, lease-based recovery that does not touch fresh locks, and permanent failures that skip retries.

Stage 2: handlers and a pool of worker threads

Handlers are ordinary functions in an object keyed by job type, so adding a job type means adding a function. They run inside worker threads, which matters for CPU-heavy jobs: a loop that counts primes would freeze the main thread (and with it the pool's own bookkeeping) if it ran there.

The pool is the only class that touches both worlds. For each slot it keeps a worker and the job currently assigned to it. Its logic:

  1. start() spawns size workers and calls #dispatch(), which gives every idle slot a claimed job (postMessage). A timer calls #dispatch() every few milliseconds, so delayed and retried jobs get picked up when they become due.
  2. When a worker replies, #finish stores the outcome (complete or fail), frees the slot, and dispatches again, so a busy queue keeps all workers busy.
  3. If a worker exits while it holds a job, the pool fails that job with worker exited with code N, spawns a replacement and carries on.

The file runs itself inside the worker: the same main.mjs is started with new Worker(import.meta.filename), and isMainThread decides which half executes. In a real project the worker code would live in its own file. Here are the handlers in the demo, and the final table the program prints:

  • uppercase and sum-primes always work (the second counts the primes up to 50,000).
  • flaky throws on its first two attempts and succeeds on the third, to show retries.
  • always-fails always throws, with only two attempts allowed.
  • crash calls process.exit(3), which in a worker ends only that worker thread, like a hard crash.
  • send-fax has no handler, so the worker answers with a permanent failure, which skips retries: retrying cannot create a missing function.
Node.js Stage 2: a worker pool processing six jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the table row by row:

  • Jobs 1 and 2 finished on the first attempt. The primes result is 5133, stored as JSON text like all results.
  • Job 3 (flaky) needed 3 attempts and is done. Its error column still holds temporary glitch on attempt 2: the last failure is kept as history even after success. That is useful when you debug "why did this take so long".
  • Job 4 (always-fails) used both attempts and is dead with its error.
  • Job 5 (send-fax) is dead after a single attempt, thanks to the permanent flag.
  • Job 6 (crash) took down its worker twice. Each time the pool noticed, recorded worker exited with code 3, and started a new worker, so jobs 1 to 5 were never affected. After two attempts it is dead.

Which worker runs which job, and in what order they finish, can vary from run to run. The program therefore waits for the queue to drain and prints the final state sorted by id, which does not vary. Never rely on completion order in your own code, and never print it in a test (see concurrency control).

Size the pool to the work. CPU-bound handlers want about one worker per core. I/O-bound handlers (waiting for HTTP or a database) are better off running as many concurrent async calls inside fewer workers, or even on the main thread with a concurrency limit. Threads are not free: each one starts its own V8 engine and uses memory.

Stage 3: persistence, crashes and graceful shutdown

Using a file instead of :memory: is what makes the queue durable. The next program simulates two process lifetimes with the same database file (jobs.db, which lives in the editor's temporary folder for this run only).

  • Process 1 enqueues four jobs and claims one with a short 50 ms lease, then "dies" without completing it.
  • Process 2 opens the same file. The dead job is still running, and calling recoverStale() at once recovers nothing, because the lease is valid: the original worker might just be slow. After the lease expires it recovers one job and sets it back to pending. Its attempt counter stays at 2, because the lost attempt counted.
  • Graceful shutdown: the pool starts and stop() is called immediately. stop() stops claiming new jobs, waits until the slots that hold a job report back, and only then terminates the workers. Two jobs finish, two stay pending. Nothing is lost and nothing is cut in half.
  • A later start processes the rest, and the queue ends with all four jobs done.
Node.js Stage 3: restart, recovery and graceful stop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The recovery design deserves a note. A lease is a promise with an expiry date: "this job is mine until time T". If the worker finishes, it completes the job. If it dies, the lease runs out and someone else may take over. This gives you at-least-once delivery: a job might run twice (the first worker was only slow, not dead) but is never lost. Exactly-once is not achievable in general, so write handlers to be idempotent: running them twice must be harmless. Typical tricks are an idempotency key stored with the effect ("welcome email sent to user 42"), or INSERT ... ON CONFLICT DO NOTHING for the side effect itself.

For graceful shutdown in a real service, connect stop() to signals:

JavaScript
// Read-only sketch for a real service.
for (const signal of ['SIGINT', 'SIGTERM']) {
  process.once(signal, async () => {
    console.log(`${signal}: finishing running jobs`);
    await pool.stop();   // no new claims, wait for the running ones
    store.close();
    process.exit(0);
  });
}

Give the shutdown a deadline, too. Orchestrators such as Docker and Kubernetes send SIGKILL after a grace period (often 10 to 30 seconds), and a job that runs longer than that will be recovered by its expired lease anyway. See process and deployment.

Where to go next

  • Priorities and queues. Add a priority column and sort ORDER BY priority DESC, run_at, id, or use separate queues (queue column) so slow reports cannot block urgent emails.
  • Heartbeats. Long jobs should extend their lease while they work (UPDATE ... SET locked_until = ?), so a healthy job is never stolen.
  • Timeouts. Terminate a worker whose job runs too long, and fail the job. You already have the respawn logic.
  • Cleanup. Delete done jobs older than a week and keep dead ones until somebody looks at them.
  • Scheduled jobs. A recurring job re-enqueues itself with delayMs when it finishes.
  • Scaling out. Several processes can share one database server and claim safely thanks to the atomic claim. SQLite in one file is for one machine. See scalability patterns.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is "SELECT the oldest pending job, then UPDATE it" an unsafe way to claim a job?

    The check and the write are separate steps, so another worker can slip in between. A single UPDATE with a subquery and RETURNING makes picking and marking one atomic step.
  2. 2. What does the lease (locked_until) protect against?

    If a worker dies, nobody completes its job. When the lease expires, recoverStale puts the job back to pending so another worker can take it.
  3. 3. A job fails on attempt 3 of 5 and the backoff base is 1000 ms. How long is the wait before attempt 4?

    The wait is backoffMs * 2 ** (attempts - 1). After the third attempt that is 1000 * 2 ** 2 = 4000 ms.
  4. 4. Why does the pool fail a job with retry: false when no handler exists for its type?

    Retries help with temporary problems such as timeouts. A permanent problem will fail every time, so retrying only wastes attempts and delays the alert.
  5. 5. What delivery guarantee does the lease-based design give, and what does it require from handlers?

    A slow worker whose lease expired may run a job that is later run again by another worker. Jobs are never lost but may run twice, so repeating a handler must be harmless.

Summary

  • A job queue moves slow or unreliable work out of the request path; the database holds the state, workers do the work.
  • Claim jobs with one atomic statement (UPDATE ... WHERE id = (SELECT ...) RETURNING). Never SELECT then UPDATE.
  • Retry with exponential backoff, cap the attempts, and keep failed jobs as dead with their last error. Skip retries for permanent errors.
  • Run handlers in worker threads, and replace workers that crash. Completion order is not deterministic; do not depend on it.
  • Leases give crash recovery and at-least-once delivery, so make handlers idempotent.
  • Stop gracefully: no new claims, wait for running jobs, then terminate workers.
  • Inject the clock so time-based logic is testable without waiting.

Next: Project: Directory Snapshot and Diff Tool