Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 133 ยท Asynchronous Programming

Concurrency Control and Queues

Control async concurrency in Node.js - limit parallel tasks with a semaphore or p-limit style pool, build a task queue, rate limit, retry with backoff and add timeouts with AbortSignal.timeout.

Starting asynchronous work is cheap, which is exactly the danger. A loop that fires 10,000 requests at once can exhaust file descriptors, trip a server's rate limit, run your database out of connections, or simply eat all your memory. On the other side, running everything one at a time wastes the time you could have saved by overlapping waits. The skill is controlled concurrency: do many things at once, but not too many.

This lesson builds the standard tools by hand, from a few lines of code: a concurrency limiter, a task queue, a rate limiter, retries with exponential backoff, and timeouts. No packages are needed (they exist, such as p-limit and p-queue, but they are small enough that you should understand them). Throughout, we avoid real clock time in the output so every run prints the same text.

The problem: unbounded parallelism

Promise.all(items.map(work)) starts every task immediately. With three items that is fine. With a million it is a bug. To see the effect safely we count how many tasks are active at the same moment.

Node.js How many run at once with Promise.all?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

All 200 ran at once. If each one opened a socket or a file, you would see errors like EMFILE: too many open files or ECONNRESET on a busy machine. We want a way to say "at most 5 at a time".

A concurrency limiter (p-limit by hand)

The idea: keep a count of running tasks and a queue of waiting ones. When a task finishes, start the next waiting one. The function below wraps any async function so it respects the limit.

Node.js limit(n): at most n tasks at a time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The peak is exactly 5, and the results still come back in input order because Promise.all keeps that order no matter when each promise settled. Notice the details a careful implementation needs: errors must free the slot too (that is the finally), and a task that throws synchronously must still become a rejection (the Promise.resolve().then(task) handles that).

A semaphore

A semaphore is the same idea as a reusable object with acquire and release. It is useful when the protected section is not a single function call, for example "hold a connection while you run three statements".

Node.js A semaphore with acquire and release

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

A and B start at once; C waits until one of them finishes, and so on. The use helper with try/finally is the key to correctness: always release in a finally, or one thrown error leaks a permit forever and the program slowly deadlocks.

A task queue

A limiter takes tasks as you call it. A queue is the same machine with a friendlier face: you add tasks over time and can wait until it is empty. This is how worker-style code (crawlers, importers, email senders) is usually organized.

Node.js A queue with concurrency and onIdle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three things to note. add returns a promise for that task's result, so callers can handle individual failures (we attached a .catch to every one; leaving it off would create an unhandled rejection). The queue never stops because of one failure. And onIdle() tells you when everything is over, which is what you await before process.exit or before closing a database.

Real queues add priorities, pause/resume, and bounded queue sizes. The last one matters: if producers are faster than consumers, an unbounded queue is a memory leak with extra steps. Make add wait (or reject) when size exceeds a threshold, so the producer feels backpressure, the same idea as in streams.

Rate limiting

A concurrency limit caps how many tasks run at once. A rate limit caps how many start per time window ("100 requests per minute"), regardless of how fast they finish. The classic algorithm is the token bucket: a bucket refills at a steady rate, each action takes a token, and an empty bucket makes the caller wait.

To keep the lesson deterministic we give the limiter a fake clock instead of reading the real time. In production you pass Date.now (or performance.now).

Node.js A token bucket with an injected clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The bucket starts full (3 tokens), so the first requests pass as a burst. Tokens also trickle in at 2 per second (one every 500 ms) while the burst is going on, which is why five requests get through before the first DENIED. From then on only one request in two is allowed, matching the refill rate of one token per two requests. Capacity controls the burst size; the refill rate controls the long-run average. In a server you would keep one bucket per client key (an IP or API token) and answer 429 Too Many Requests on a denial.

Retries with exponential backoff

Networks fail in short, temporary ways. Retrying is right for transient errors (timeouts, ECONNRESET, HTTP 503, 429) and wrong for permanent ones (a 404 or a validation error will fail forever). Wait longer after each failure (exponential backoff: 100 ms, 200 ms, 400 ms...), and add jitter (a random fraction) so that many clients do not retry in lock-step and hit the server together.

Node.js Retry with backoff and jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With sleep and random passed in, the function is easy to test: no real waiting, no randomness. The delays here are 100 ms after the first failure and 200 ms after the second. In production use timers/promises for sleep and Math.random for random, cap the delay (for example at 30 seconds), and always give the whole operation an overall deadline. Retries multiply load: if every layer of your system retries 3 times, a failure at the bottom becomes 27 requests. Retry in one place, and respect Retry-After headers when a server sends them.

Only retry operations that are idempotent (safe to run twice), such as a GET or a PUT of the same data. Retrying "charge the credit card" after a timeout can charge twice, because the first attempt may have succeeded. Use idempotency keys for such operations.

Timeouts with AbortSignal.timeout

Every operation that waits on something outside your control needs a deadline. AbortSignal.timeout(ms) creates a signal that aborts itself after ms milliseconds, with a TimeoutError. Pass it to any API that accepts a signal (fetch, fs functions, timers/promises, streams), and the operation is actually cancelled, not merely abandoned. Cancellation is explored fully in AbortController and Cancellation.

Node.js A deadline with AbortSignal.timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first call throws before the 200 ms sleep ends, with an AbortError whose cause is the signal's TimeoutError (what you see depends on the API: fetch rejects with the TimeoutError itself); because sleep received the signal, its timer is really cleared. For code that does not accept a signal, Promise.race against a timeout only stops waiting, while the slow work keeps running in the background and holds its resources. Prefer a signal-aware API whenever one exists.

Work pools and when to use what

Situation Tool
Run a list of async jobs, at most N at once limiter / p-limit style function
Jobs arrive over time, need onIdle, priorities task queue (p-queue)
Protect a resource with a fixed number of slots semaphore
Stay below an API's requests per minute token bucket / rate limiter
Temporary failures retry with backoff and jitter
Any call to the outside world timeout (AbortSignal.timeout)
CPU-heavy jobs a pool of worker threads

Asynchronous concurrency only helps with waiting (I/O). A promise limiter does nothing for CPU-heavy work, since all the JavaScript still runs on one thread; for that you need worker threads. A good starting number for I/O limits is whatever the downstream system allows: database pool size, an API's documented concurrency, or about 10 to 50 for HTTP calls. Measure, do not guess.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main risk of Promise.all(items.map(task)) for a huge array?

    Promise.all does not limit how many tasks run at once, because map starts every one immediately.
  2. 2. Why must a semaphore permit be released in a finally block?

    If an error skipped the release, the permit would be lost forever and eventually every acquire would wait for ever.
  3. 3. What is the difference between a concurrency limit and a rate limit?

    Concurrency limits bound simultaneous work; rate limits bound the speed at which work starts, whatever its duration.
  4. 4. Which operation is safe to retry automatically after a timeout?

    A GET is idempotent: doing it twice gives the same result and no side effects. The other operations can be applied twice.
  5. 5. What advantage does AbortSignal.timeout have over racing a promise against a timer?

    A signal passed to the API stops the underlying work, while a lost Promise.race only stops waiting for it.

Summary

  • Starting everything at once with Promise.all(map(...)) is unbounded; cap concurrency deliberately.
  • A limiter counts running tasks and queues the rest; always free the slot in finally.
  • A semaphore and a task queue are the same machinery with different interfaces; bound queue length to avoid memory growth.
  • A token bucket limits the starting rate and allows controlled bursts; inject the clock so you can test it.
  • Retry only transient errors on idempotent operations, with exponential backoff, jitter and a cap.
  • Give every outside call a deadline with AbortSignal.timeout, and pass the signal on so work is really cancelled.
  • Promise-based limits help with I/O waiting; use worker threads for CPU-bound jobs.

Next: AbortController and Cancellation