Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 133 ยท Processes and Threads

Cluster, Scaling and Load Balancing

Scale Node.js beyond one CPU core. Learn the cluster module, round-robin balancing, sticky sessions, PM2, containers and why stateless, shared-nothing design makes horizontal scaling work.

A Node.js process runs your JavaScript on one thread, and one thread can use only one CPU core. A server with 8 cores running a single Node process leaves seven of them idle. To use the machine fully, and to survive crashes, you run several copies of the program and spread incoming requests across them. That is scaling, and the way you do it shapes how you write the application.

There are two directions. Vertical scaling means a bigger machine (more cores, more RAM). Horizontal scaling means more machines or more processes behind a load balancer. This lesson starts with the built-in node:cluster module, which forks several worker processes that share one server port. It then covers load balancing, sticky sessions, process managers such as PM2, containers, and the design rule that matters most: keep your servers stateless, so any copy can answer any request.

Our sandbox has one CPU core, so you will not see a speed-up here. What you can see, on loopback, is how requests are distributed, how crashed workers are replaced, and what breaks when state lives inside a worker.

What the cluster module does

cluster builds on child processes: the first process (the primary, formerly called the master) calls cluster.fork() to start workers, each a full copy of your program in its own process, connected to the primary by an IPC channel. The magic is in server.listen(). When a worker listens on a port, the request is forwarded to the primary, which owns the real listening socket and hands out incoming connections to the workers. All workers appear to listen on the same port without clashing.

The same file runs in both roles. cluster.isPrimary tells you which one you are:

Node.js Three workers sharing one port

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

All three workers asked for port 0 ("any free port") but ended up on one port, because the primary created the socket and shares it. Nine requests were spread evenly: three per worker.

How the primary chooses a worker is the scheduling policy:

  • Round robin (cluster.SCHED_RR, the default on Linux and macOS): the primary accepts each connection and passes it to the next worker in turn. The distribution is even.
  • OS-driven (cluster.SCHED_NONE, the default on Windows): the workers share the socket and the operating system decides. It is sometimes badly unbalanced, with most connections landing on one worker. If you run the example on Windows, you will likely see a different spread.

Set it with cluster.schedulingPolicy = cluster.SCHED_RR before forking, or the environment variable NODE_CLUSTER_SCHED_POLICY (rr or none).

Balancing happens per connection, not per request. A browser or an HTTP client with keep-alive sends many requests down one connection, and they all go to the same worker. That is why the example sends connection: close. In production, a long-lived connection can keep one worker busy while others idle.

Staying alive: restarting crashed workers

The second big benefit of several processes is resilience. If a worker crashes on a bug or runs out of memory, the others keep serving. The primary watches for the 'exit' event and starts a replacement. This is the core loop of every process manager.

Node.js The primary restarts a crashed worker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice the restart limit. Without it, a worker that crashes at start-up would be restarted in an endless, CPU-burning loop (a "crash loop"). Real supervisors add a delay that grows after each failure ("backoff") and alert you when the limit is hit. For more on crash-and-restart policies, see Error Handling in Production.

Rolling restarts

To deploy new code without dropping requests, restart workers one at a time: call worker.disconnect(), which stops it accepting new connections and lets it finish current ones, wait for its 'exit', and then fork the replacement before moving to the next worker. Because the primary keeps the listening socket, clients never see "connection refused".

The real problem: state

Running several copies is easy. Writing an application that still behaves correctly with several copies is the hard part. Each worker is a separate process with its own memory. A variable in one worker is invisible to the others.

Node.js In-memory state breaks with several workers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With round robin the login lands on one worker, and later requests alternate between that worker (which knows the user) and the other (which does not). The output shows both ada and nobody among the five answers, and the two includes lines print true. (Which answer comes first depends on which worker the round robin starts with.) This is the classic "I log in and am randomly logged out" bug that appears the moment an app is scaled out. The same trap catches in-memory caches, rate-limit counters, WebSocket rooms and uploaded-file temp state.

There are three ways out, from best to least pleasant:

  1. Make servers stateless. Keep nothing important in process memory. Put sessions in a shared store (Redis, a database), or in a signed cookie or token the client sends back with every request (see Headers, Cookies and Caching). Any worker, or any server, can then answer any request. This is the foundation of horizontal scaling.
  2. Use sticky sessions. Make sure the same client always reaches the same worker (next section). Easier to adopt, but it limits balancing and loses data if that worker dies.
  3. Share state through the primary over IPC. It works for tiny cases but makes the primary a bottleneck. Use a real shared store instead.

Load balancing and sticky sessions

A load balancer sits in front of several servers and forwards each request to one of them. The rule it uses to choose is the algorithm:

Algorithm Idea Good for
Round robin Next server in turn Equal servers, similar requests
Least connections Server with the fewest active connections Requests with very different durations
Hash (IP or cookie) Same client key always maps to the same server Sticky sessions, caches
Weighted Bigger servers get more Mixed hardware

Here is a miniature load balancer: two backends and a proxy in front of them. It routes requests either round-robin or by hashing a session id, and you can watch what each approach does to a single user.

Node.js Round robin versus sticky sessions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With round robin the same user's four requests alternate between backend A and backend B (the number after # is how many requests that backend has served so far). With the hash strategy, the session id user-42 always maps to the same backend, so one backend answers all four. The counter continues from the round-robin run (A#3 to A#6) because each backend keeps its own count of everything it has served. The price of stickiness is uneven load (a popular user pins one server) and lost sessions when a server restarts or is removed.

Real balancers do the same with a cookie or the client IP: nginx ip_hash, a cloud load balancer's "session affinity", HAProxy stick-table. Prefer stateless design and use stickiness only for things that truly need it, such as WebSocket fallbacks that need several requests to reach the same server (Socket.IO's long-polling transport is the well-known example).

Process managers: PM2 and friends

You rarely write the restart loop yourself in production. A process manager starts your app, restarts it when it dies, collects logs and can run it in cluster mode. The most common for Node is PM2. These snippets run on your own server, not in our editor:

Terminal
# Read-only: PM2 on a server
npm install --global pm2
pm2 start server.js -i max       # one worker per CPU core, using cluster
pm2 list                          # status of all processes
pm2 reload server                 # zero-downtime restart, one worker at a time
pm2 logs
pm2 startup && pm2 save           # start automatically after a reboot
JavaScript
// Read-only: ecosystem.config.cjs
module.exports = {
  apps: [{
    name: 'api',
    script: './server.js',
    instances: 'max',
    exec_mode: 'cluster',
    max_memory_restart: '300M',
    env: { NODE_ENV: 'production' },
  }],
};

Alternatives: systemd (the Linux init system can restart a service on failure), and the container platforms below. If you already run containers, a process manager inside the container is usually redundant.

Containers and horizontal scaling

Modern deployments often run one Node process per container and let the platform handle replication:

DOCKERFILE
# Read-only: Dockerfile fragment
FROM node:24-slim
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
CMD ["node", "server.js"]
YAML
# Read-only: Kubernetes Deployment excerpt
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: api
          image: example/api:1.0.0
          resources:
            requests: { cpu: "500m", memory: "256Mi" }
          readinessProbe:
            httpGet: { path: /ready, port: 3000 }

The platform's load balancer spreads traffic over the 4 replicas, replaces crashed ones, and an autoscaler adds replicas under load. In this model you do not use cluster. A container limited to one CPU gains nothing from it, and the orchestrator already does restarts. Use cluster or PM2 when you run directly on a multi-core server, without containers.

Shared-nothing design

Whether you scale with cluster, PM2, Docker or Kubernetes, the winning design is shared-nothing: every instance is identical, disposable and independent.

  • No local state that matters. Sessions, carts, job queues and caches that must be consistent live in a database or a shared store, not in a variable.
  • Configuration from the environment. Every copy reads the same variables (see Configuration, Secrets and Environment).
  • Local files are temporary. Uploads go to object storage, not a worker's disk.
  • Idempotent work and queues. If two copies might do the same job, make doing it twice harmless, or use a queue that hands each job to exactly one worker.
  • Graceful shutdown. On SIGTERM, stop accepting connections, finish in-flight requests, then exit (see Graceful Shutdown and Health Checks).
  • Health checks. Expose /health (alive) and /ready (ready for traffic) so the balancer sends no requests to a starting or broken instance.

A cache in memory is still useful as a per-instance optimisation (it is only a hint that may be absent or stale), but it must never be the source of truth.

Cluster, workers or more servers?

  • Need to use all cores for an HTTP server on one machine, without containers: cluster or PM2 cluster mode.
  • A heavy computation inside a request: keep one server process and offload to worker threads.
  • Different languages or programs: child processes behind your Node server.
  • Traffic beyond one machine, or a need for redundancy: several servers behind a load balancer, stateless.

Before scaling, measure. A slow endpoint caused by an N+1 query or a blocked event loop will be just as slow on eight copies. See Measuring Performance and Performance Tuning.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How can several cluster workers all listen on the same port?

    server.listen() in a worker is forwarded to the primary, which creates the real socket and distributes incoming connections to the workers.
  2. 2. Why does a session stored in a JavaScript variable break when you run several cluster workers?

    Workers share nothing by default. A request that reaches another worker finds no session. The fix is a shared store, a signed cookie or sticky routing.
  3. 3. What is a drawback of sticky sessions?

    Pinning a client to one server limits balancing, and the client's state disappears when that server is restarted or removed.
  4. 4. Why should the primary limit how often it restarts a crashing worker?

    A worker that fails at start-up would otherwise be restarted forever. Supervisors add limits and growing delays and alert someone.
  5. 5. You deploy one Node process per container on Kubernetes, each limited to one CPU. Is cluster useful?

    With one CPU per container, extra processes just compete. The orchestrator scales by adding replicas and restarting failed ones.

Summary

  • One Node process uses one CPU core. cluster forks workers that share a port through the primary; the default policy is round robin on Linux and macOS and OS-driven on Windows.
  • Balancing is per connection, so keep-alive connections stay on one worker.
  • The primary should restart crashed workers, with a limit and backoff, and use worker.disconnect() for rolling restarts.
  • Process memory is not shared. Keep servers stateless and move sessions, caches that must agree, and queues to shared stores; use sticky sessions only when you must.
  • PM2 and systemd supervise processes on a server; containers and orchestrators replicate instances in cloud setups.
  • Design shared-nothing, add health checks and graceful shutdown, and measure before you scale.

Next: Measuring Performance