Skip to content
Tutorials Center
My learning

Topics

Lesson 108 of 133 ยท Production Node.js

Graceful Shutdown and Health Checks

Learn graceful shutdown in Node.js - handle SIGTERM, drain HTTP connections, close resources in order, set shutdown timeouts, and design liveness and readiness checks.

Servers do not run forever. Every deploy replaces old processes with new ones, autoscalers remove machines, and operators press Ctrl+C. Each time, your process is told to stop while it may be in the middle of ten requests, a database transaction and a half-written file. If it simply dies, those requests fail, clients see errors, and data can be left in an inconsistent state. A graceful shutdown is the polite alternative: stop taking new work, finish the work in progress, release resources, and then exit.

This lesson covers the whole story: how the operating system asks a process to stop (signals), what a correct shutdown sequence looks like for an HTTP server, how to set a deadline so shutdown cannot hang, and how health checks tell a load balancer when to send you traffic and when to stop. All examples run on the loopback interface so you can watch them work.

How processes are asked to stop

On Linux and macOS, the operating system and process managers communicate with your program through signals. The ones that matter for shutdown:

Signal Sent when Can you handle it?
SIGINT You press Ctrl+C in a terminal Yes
SIGTERM A deploy, kill <pid>, docker stop, Kubernetes pod termination Yes
SIGHUP The terminal closed (sometimes used to reload config) Yes
SIGKILL The manager gives up waiting, or the kernel kills you for using too much memory No

SIGTERM means "please finish and exit". If you do not exit in time, the manager follows up with SIGKILL, which cannot be caught, delayed or cleaned up after. Docker waits 10 seconds by default, Kubernetes 30 seconds (the pod's termination grace period). So a graceful shutdown has a budget, and you must finish inside it.

In Node you listen with process.on('SIGTERM', handler). The signals themselves are covered in The process Object; here is the minimum. A process can even send a signal to itself, which is handy for a demonstration.

Node.js Receiving a signal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Without a handler, SIGTERM terminates the process immediately. Once you register one, Node no longer exits by itself: you are responsible for calling process.exit() (or for letting the event loop drain). That is the power and the danger of the handler: if you register it and forget to exit, the process ignores every polite request to stop until it is killed.

On Windows, only a few signals exist. Ctrl+C produces SIGINT, and closing the console produces SIGBREAK, but SIGTERM is not delivered by taskkill. Production servers almost always run on Linux, so write for signals and test on Linux (or in a container).

What goes wrong without a graceful shutdown

Let us make the problem visible. The program below starts a small server as a child process. The server handles a slow request that takes 300 ms. While that request is in flight, the parent sends SIGTERM. We run it twice: once with a naive handler that exits immediately, and once with a graceful handler that stops listening, waits for active requests and then exits.

Node.js Naive exit versus graceful exit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

In the naive run the customer's checkout request died (fetch failed), the server's log stops at the signal line, and the exit code is still 0, so nothing on the server side reveals that a customer was hurt. In the graceful run the client received order processed, and the server logged that it had drained and exited with code 0. The only difference is the three lines in the handler.

The shutdown sequence

Order matters. A reliable sequence for a typical service:

  1. Mark yourself as shutting down. From now on, readiness checks report "not ready" so the load balancer stops sending new traffic.
  2. Wait briefly (optional, a few seconds) so that the load balancer and service discovery notice. Without this pause, a request already routed to you may arrive just after you stop listening.
  3. Stop accepting new connections: server.close(). Existing connections stay open.
  4. Drain: let in-flight requests finish. Close idle keep-alive connections so they do not hold up close().
  5. Close dependencies in reverse order of creation: HTTP clients, queue consumers, then the database pool, then the cache connection, then flush logs and metrics.
  6. Exit with code 0 on success, non-zero if you had to force it.
  7. A timer runs from the first signal: if the sequence takes longer than your budget, log it and exit with a failure code anyway.

Why server.close() alone is not enough

server.close() stops the listening socket. Its callback fires only when every connection has ended. Browsers and fetch use HTTP keep-alive, so a connection often stays open and idle after its response, waiting for the next request. Node's default keep-alive timeout is 5 seconds, so a plain close() can seem to hang for several seconds even though no request is running.

The server API has two helpers: server.closeIdleConnections() closes sockets that are not currently handling a request, and server.closeAllConnections() closes everything, including active requests (a last resort used after the timeout). The idle sweep must be repeated, because a connection that was busy during the first sweep becomes idle after its response. That is why the example above sets connection: close on responses sent during shutdown: each connection ends right after its final response.

A reusable shutdown manager

Hand-writing the sequence in every service leads to bugs. A small manager that runs registered cleanup steps in the right order, with a timeout and protection against running twice, is easy to build. Cleanup steps register themselves in the order resources are created, and run in reverse (last in, first out), exactly like nested try/finally blocks.

Node.js A shutdown manager with ordering and a timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The steps ran as http server, message consumer, database pool: the opposite of registration. That is deliberate: the HTTP server stops first so no new request needs the database, and the pool closes last. The second run call was ignored. In the second scenario one step never finishes (think of a client that never closes a long-lived connection), so the timer forces an exit with code 1 after 100 ms and the database pool step never gets its turn. A real system would also try each remaining step; the point of the timeout is that something ends the process before SIGKILL does it for you.

Two details deserve attention. The timer calls unref() so it does not keep an otherwise finished process alive. And the manager calls an injected onForceExit instead of process.exit directly, which makes it testable (see Mocks, Spies and Fake Timers). In the real service you pass process.exit.

Long-lived connections

HTTP/1.1 requests are short, but servers also hold WebSockets (see WebSockets), server-sent events (Server-Sent Events) and long downloads. server.close() waits for these forever. Your shutdown must actively tell them to leave: send a close frame with code 1001 ("going away") to WebSocket clients, end event streams with a final message that tells the browser to reconnect (to another instance), and abort downloads that exceed a grace period. Clients should reconnect automatically with backoff, as in Error Handling in Production.

The same applies to background work. A queue consumer should stop pulling new jobs at once and finish (or hand back) the current one; a scheduled task should not start a new run. If a job takes longer than your grace period, make sure it is safe for another worker to retry it. Idempotent jobs are the safety net under every shutdown.

Health checks: liveness and readiness

Platforms decide what to do with your process by calling HTTP endpoints you provide. Three questions, three different endpoints:

  • Liveness ("are you stuck?"): answers 200 whenever the event loop is running. If it fails, the platform restarts the process. Keep it trivial. Never check the database here: a database outage would make the platform restart all your healthy processes in a loop, making things worse.
  • Readiness ("can you serve traffic right now?"): answers 200 when the app is initialized and its critical dependencies work, and 503 when it should be taken out of rotation. A failing readiness check does not restart the process; the load balancer just stops sending requests. During shutdown, readiness must flip to 503 first.
  • Startup ("have you finished booting?"): protects slow starts (loading caches, warming connections) so liveness does not kill the process too early.

Typical paths are /livez and /readyz (or /healthz). The response body can be tiny; platforms read the status code.

Node.js Liveness and readiness endpoints through a shutdown

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the table of results from top to bottom. While booting, the process is alive but not ready. When the database goes down, only readiness changes; the process keeps running and rejoins the pool on its own when the database returns. During shutdown, readiness fails while liveness still passes: the platform must not restart a process that is busy shutting down cleanly.

Keep readiness checks cheap. A check that runs a heavy query every few seconds for every instance becomes a load problem of its own. Cache the result for a second or two, or test the connection pool's state instead of running a query.

Putting it together in a real service

A complete wiring for a service looks like this (read-only sketch, using the pieces above):

NODE
const shutdown = createShutdown({ timeoutMs: 25_000, onForceExit: process.exit });

shutdown.add('database pool', () => pool.end());
shutdown.add('http server', async () => {
  state.shuttingDown = true;                       // readiness now answers 503
  await new Promise((resolve) => setTimeout(resolve, 5_000)); // let the load balancer notice
  await new Promise((resolve, reject) => {
    server.close((error) => (error ? reject(error) : resolve()));
    server.closeIdleConnections();
  });
});

for (const signal of ['SIGTERM', 'SIGINT']) {
  process.on(signal, () => shutdown.run(signal));
}

Choose the timeout below the platform's grace period (here 25 s against Kubernetes' 30 s), so you exit on your own terms with a log line instead of being killed in silence.

Pitfalls professionals know

  • npm start as PID 1. In a container, if the entrypoint is npm start or a shell script, the signal goes to npm or the shell, which may not forward it, and your handler never runs. Start node server.js directly (exec form in the Dockerfile: CMD ["node", "server.js"]) and consider docker run --init for correct zombie reaping.
  • Handlers registered too late. A signal that arrives before process.on('SIGTERM') runs kills the process the old way. Register handlers early in startup.
  • Registering handlers inside request code. Each registration adds another listener; you end up with dozens and a warning. Register once.
  • Calling process.exit() too early. It terminates immediately, abandoning pending writes to stdout, files and sockets. Prefer a controlled sequence, set process.exitCode, and call process.exit only as the final step or the forced path.
  • Two signals. Operators press Ctrl+C twice when impatient. Make the first start the graceful sequence and let a second one force exit.
  • Forgetting the logs. Flush your logger before exiting, or the most important lines (the shutdown reasons) are lost.
  • Tests that never exit. The same mistakes (a server never closed) that hang production shutdown hang test runs too; see Testing HTTP Servers and CLIs.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which signal can a Node.js process NOT catch or handle?

    SIGKILL is handled by the kernel and cannot be caught. It is what the platform sends when a process ignored SIGTERM past the grace period.
  2. 2. Why can server.close() seem to hang even when no request is running?

    The close callback waits for every connection to end. Idle keep-alive sockets stay open, so call server.closeIdleConnections() or send connection: close on final responses.
  3. 3. During shutdown, what should the readiness endpoint return, and the liveness endpoint?

    Readiness must fail first so the load balancer stops routing traffic. Liveness stays green so the platform does not restart a process that is shutting down on purpose.
  4. 4. Why should liveness checks avoid querying the database?

    Restarting the app does not fix a database outage. Dependency checks belong in readiness, which only removes the instance from rotation.
  5. 5. Your platform gives pods 30 seconds before SIGKILL. What should your own shutdown timeout be?

    Your forced-exit timer must fire before the platform kills you, otherwise you lose the chance to log what hung and exit with a clear code.

Summary

  • SIGTERM (deploys, docker stop) and SIGINT (Ctrl+C) ask for a clean exit; SIGKILL cannot be handled and follows when you take too long.
  • A signal handler makes you responsible for exiting: stop accepting work, drain, close resources, then exit.
  • server.close() waits for all connections; use closeIdleConnections(), connection: close on final responses, and closeAllConnections() only as a last resort.
  • Close dependencies in reverse order of creation, protect against running shutdown twice, and enforce a timeout shorter than the platform's grace period (with an unref'd timer).
  • WebSockets, event streams and background jobs need explicit shutdown handling.
  • Liveness says "restart me", readiness says "send me traffic"; readiness fails first during shutdown, and neither should be heavy.
  • In containers, run node directly as the entrypoint so signals reach your handler.

Next: Processing Large Data Efficiently