Skip to content
Tutorials Center
My learning

Topics

Lesson 110 of 133 ยท Production Node.js

Scalability and Reliability Patterns

Build Node.js services that scale and survive failure - stateless design, caching with single-flight, token bucket rate limiting, idempotency keys, timeouts, load shedding and queues.

A service that works for ten users and a service that works for ten million are usually the same code with different habits around it. Scalable systems share a handful of patterns: keep servers interchangeable, answer repeated questions from a cache, slow down greedy clients, make retries safe, never wait forever, refuse work you cannot finish, and move slow work behind a queue. None of these needs a framework. Each is a few dozen lines of Node.js, and in this lesson you will build every one of them and run it.

The lesson assumes you know the http module, promises and AbortController. It pairs well with Error Handling in Production and Caching Strategies.

Scale out, not just up

You can make one server bigger (vertical scaling) or run many copies behind a load balancer (horizontal scaling). Node is single-threaded for JavaScript, so a copy uses one core; to use a 16-core machine you run several processes with cluster or a process manager. Horizontal scaling only works if copies are stateless: any request can go to any copy, and no copy remembers anything between requests that another copy would need.

That rules out some convenient habits:

  • Sessions kept in a JavaScript Map inside the process. Use a shared store (a database or Redis) or signed tokens.
  • Files written to the local disk and read back later. Use object storage or a database.
  • In-memory counters for rate limits or job state that other copies must see.
  • Scheduled work that every copy runs, so it happens N times. Use one scheduler or a lock.

Small, per-process caches that are only an optimization are fine: if a copy loses its cache it just recomputes. The rule is never put the only copy of important data in process memory. The rest of the lesson shows patterns that keep one process honest, with a note where a shared store would replace the in-process version.

Caching with single-flight

A cache answers repeated questions without redoing the work. The simplest is a Map with an expiry time. The subtle problem is the cache stampede: when a popular entry expires, a hundred concurrent requests all miss, all run the expensive query, and the database falls over. The cure is single-flight: store the promise of the pending load, so concurrent callers share one load.

The example uses a fake clock, so the result never depends on real time.

Node.js A TTL cache with single-flight loading

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice what is not cached: a failed load. If you stored the error, one blip would break the key for the full TTL. Three more production notes: add a maximum size (evict the oldest or least-used entries) or a busy cache eats all memory; add a little random jitter to TTLs so thousands of keys do not expire at the same instant; and remember a process-local cache gives each copy its own view, so data may be briefly stale in different ways on different servers. For one shared cache use Redis or Memcached, with the same single-flight idea in front of it.

Rate limiting with a token bucket

A rate limiter protects you from greedy clients and runaway scripts. The token bucket algorithm is the usual choice: a bucket holds up to capacity tokens and is refilled at a steady rate. Each request takes a token; with an empty bucket the request is refused (HTTP 429 Too Many Requests). It allows short bursts (up to the capacity) while capping the long-run average rate.

The refill is computed lazily from the elapsed time when a request arrives, so there is no timer per client.

Node.js Token bucket limiter with a controllable clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Alice's burst of three succeeds and the next two are refused. After two seconds she has two tokens again, so two of three pass. Bob has his own bucket. In production you also send Retry-After and RateLimit-* headers so well-behaved clients can back off, and you keep the buckets in a shared store (an atomic Redis script is the classic choice) so the limit holds across all server copies. Delete idle buckets so the Map does not grow forever.

Idempotency keys: make retries safe

Networks fail in the worst place: after the server did the work but before the client heard about it. The client retries, and the customer is charged twice. An idempotent operation gives the same result no matter how many times it is applied. GET, PUT and DELETE already are by definition; POST is not.

The fix is an idempotency key. The client generates a unique id per intended operation and sends it in a header on every attempt. The server stores the first result under that key and returns the stored result for repeats instead of redoing the work.

Node.js Idempotency keys on a loopback payments endpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The retry gets 200 with the original charge id and no second charge. Real implementations must also handle two concurrent requests with the same key (store the key in a database row with a unique constraint, or use the single-flight idea above) and should compare the request body, rejecting a key reused with different data. Keep keys for a day or so, then expire them.

Timeouts everywhere, and retries with limits

A call without a timeout can wait forever, and waiting calls pile up: memory, sockets, and finally your own callers' timeouts. Every outgoing call (HTTP, database, queue) needs a deadline, and the deadlines should shrink as you go deeper, so a caller with a 3-second budget never calls something with a 10-second timeout. AbortSignal.timeout(ms) gives you this for fetch, streams and many other APIs.

Retries help with brief failures but can make an outage worse: if every client retries immediately, a struggling server gets double the load. Retry only idempotent calls, cap the attempts, and wait with exponential backoff plus jitter. For deep failures use a circuit breaker that stops calling a broken dependency for a while.

Load shedding: say no early

When demand exceeds capacity, doing everything slowly is the worst outcome: every request times out and no one is served. Load shedding means refusing some requests quickly (503 Service Unavailable) so the ones you accept finish on time. The simplest policy caps the number of requests in flight.

The next program has a handler that takes a moment, a limit of two concurrent requests, and a client that sends five at once with a 2-second timeout. Which requests succeed depends on arrival order, so the program sorts and counts the statuses.

Node.js Capping in-flight requests and using client timeouts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two requests were accepted and three were turned away within microseconds, each with Retry-After. The hanging call ended with TimeoutError instead of waiting forever. Typical shedding signals are in-flight count, queue length, or event loop delay. Shed low-priority traffic first (analytics before checkout) and make sure health checks stay cheap.

Queues and event-driven design

Some work does not need to finish before you answer: sending email, resizing an image, calling a slow partner. Answer 202 Accepted, put a job on a queue, and let a separate worker process it. The queue absorbs traffic spikes (a backlog is better than an outage), lets you scale workers independently, and lets you retry failures without bothering the user. In production the queue is durable (Redis streams, RabbitMQ, SQS, a database table); the shape of the code is the same as this in-process version with bounded concurrency and retries.

Node.js An in-process job queue with concurrency and retries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The flaky address succeeds on its third attempt; the broken one is abandoned after three. In a real system the abandoned job goes to a dead letter queue for a human or a repair script to inspect, rather than vanishing. Design jobs to be idempotent, since a queue usually delivers at least once, so a job may run twice.

The same decoupling applies inside a codebase: instead of the checkout code calling email, inventory and analytics directly, it emits an order.placed event and each subscriber reacts. Each part can then fail, scale and be deployed on its own. Start with a plain EventEmitter, and move to a message broker when subscribers live in other services.

Do not reach for all of these on day one. Measure first. A single well-tuned Node process behind a load balancer handles a surprising amount of traffic. Add a pattern when you can name the failure it prevents.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What makes a Node.js service easy to scale horizontally?

    Horizontal scaling adds interchangeable copies behind a load balancer. State that only one copy remembers breaks that interchangeability.
  2. 2. What problem does single-flight loading solve in a cache?

    Storing the pending promise lets concurrent callers share one load instead of each hitting the database.
  3. 3. A token bucket has capacity 5 and refills 1 token per second. A client has been idle for a minute and then sends 8 requests at once. How many succeed?

    The bucket never holds more than its capacity, so an idle client gets a burst of at most 5; the other 3 are refused until tokens refill.
  4. 4. Why send an Idempotency-Key header with a POST that creates a payment?

    If the client never saw the first response, it retries with the same key, and the server answers from the stored result, so the side effect happens once.
  5. 5. Why is load shedding often better than accepting every request?

    Past capacity, slow responses for everyone are worse than a quick 503 for some, because clients time out, retry and add even more load.

Summary

  • Keep servers stateless and interchangeable; keep only disposable caches in process memory.
  • Cache with a TTL, a size limit, jitter and single-flight loading, and never cache failures.
  • Rate-limit with a token bucket per client, return 429 with Retry-After, and share the buckets across copies in production.
  • Use idempotency keys so retries cannot repeat side effects.
  • Put a deadline on every outgoing call, retry only idempotent work with backoff and jitter, and cap attempts.
  • Shed load early with a cap on in-flight work, and push slow work behind a queue with retries and a dead letter queue.

Next: Performance Tuning