Skip to content
Tutorials Center
My learning

Topics

Lesson 107 of 133 ยท Production Node.js

Error Handling in Production

Learn production error handling in Node.js - operational vs programmer errors, central handlers, uncaughtException policy, retries, circuit breakers and dead letter queues.

On your laptop, an error is a red message you read and fix. In production, an error happens at 3 a.m., to a customer you cannot talk to, while a hundred other requests are in flight. Good production error handling answers four questions for every failure: Who needs to know? What does the user see? Can the operation be retried? Is the process still safe to keep running?

You already know how to throw, catch and wrap errors from Errors and Error Handling. This lesson is the next level: designing an error vocabulary for a whole application, handling errors in one central place, deciding what to do when the process itself is in doubt, and surviving failing dependencies with retries, circuit breakers and dead letter queues. Everything here runs in the editor, using a simulated "flaky" dependency and fixed clocks instead of real outages.

Two kinds of errors, two kinds of reactions

The most useful distinction in production is between operational errors and programmer errors.

  • An operational error is a problem that can happen in a correct program: a user sends invalid JSON, a database connection is refused, a file is missing, an upstream API times out. The code is fine; the world is not. You handle these: return a 4xx or 5xx status, retry, fall back, log a warning.
  • A programmer error is a bug: calling a function with the wrong argument type, reading a property of undefined, a broken invariant. No amount of retrying fixes it. After one, the program may be in a state nobody planned for. You fix the code, and in the meantime you log loudly and, usually, restart.

Mixing them up causes the two classic failures: swallowing a bug because you caught everything (the bug lives on silently), and crashing a whole server because one user sent a bad form (a bug in your catch policy).

A practical way to separate them is to give your own operational errors a shared base class that carries a stable machine-readable code, a safe-to-show message, an HTTP status and a flag. Everything that is not an instance of it is treated as unexpected.

Node.js An application error hierarchy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Note the expose flag. The message of a ValidationError ("email is invalid") helps the user. The message of a DependencyError or a TypeError can leak internals: host names, SQL, file paths. Decide per class, once, what is safe to show, and let a central function enforce it.

One central error handler

If every route has its own try/catch that formats its own error response, you get a dozen slightly different formats, and one route will eventually leak a stack trace. Instead, let errors propagate to one function that translates them into responses and log entries.

The handler needs to do four things: log the error with enough context, choose the status code, build a response body that never exposes internals for unexpected errors, and avoid crashing when it is itself called twice or after headers were sent. Here it is in a real HTTP server on loopback, with a small route wrapper that sends any thrown error, sync or async, to the central handler.

Node.js A central error handler in an HTTP server

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Compare the two views. The client saw Something went wrong for the bug in /users/99, with no stack, no property names. The log recorded the real TypeError message, so you can find it. Expected failures were logged at warn, the bug at error, so your alerting can page someone for the second kind only.

The res.headersSent check matters. If a handler already started streaming a response and then fails, you cannot change the status code any more. Destroying the connection tells the client the response is incomplete, which is more honest than ending it with a truncated body that looks valid.

Never show users internals, always give them a handle

Users do not need your stack trace; they need to know whether to fix their input, wait or contact support. Three practices help:

  • Expose a stable error code (VALIDATION_FAILED), which a front end can translate, rather than relying on message text.
  • For unexpected errors, return a generic message plus a request id the user can quote. Support then finds the exact log line. The request id can be generated per request (see Logging and Observability for correlation ids).
  • Never send error.stack, SQL text, file paths or environment values in a response, in any environment that real people can reach.

What happens to errors nobody caught

Some errors escape every try/catch: a throw inside a timer callback, a rejected promise nobody awaited, an EventEmitter that emits 'error' with no listener. Node turns them into two process events, and your policy for them is a safety decision.

  • process.on('uncaughtException', handler)
  • process.on('unhandledRejection', handler) (since Node 15, an unhandled rejection raises an uncaught exception by default and exits)

The tempting policy is "log it and keep running". It is also the dangerous one. After an uncaught exception, you do not know which half-finished operation was abandoned, which lock is still held or which object is half-updated. The recommended policy is: log the error, flush the logs, then exit with a non-zero code and let a supervisor restart the process (systemd, Docker, Kubernetes, PM2). Restarting is cheap in Node, and a fresh process is a known-good state. Combine this with graceful shutdown so in-flight requests get a moment to finish (see Graceful Shutdown and Health Checks).

You can see the policy in action by running a script as a child process and watching its exit code. The child below registers a last-resort handler, writes one structured line to standard error, and exits with code 1.

Node.js Crash-and-restart policy in a child process

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The child announced itself, logged a single JSON line describing the unhandled rejection, and exited with code 1 before the one-second timer could print anything. A supervisor reading exit code 1 would start a fresh process.

Do not "recover" from uncaughtException by continuing to serve traffic. The Node.js documentation itself says the event is a last resort for synchronous cleanup before shutting down. The handler's job is to log and exit, nothing more. For unhandledRejection the same reasoning applies: it means a promise failure was never handled, which is a bug.

Retrying operational errors

Many operational errors are transient: a dropped connection, a 503, a deadlock the database aborted. Trying again a moment later often works. Retrying is only safe when the operation is idempotent (doing it twice has the same effect as once) or when you use an idempotency key, so never blindly retry a payment charge.

A good retry policy has four ingredients:

  1. A maximum number of attempts, so a permanent failure ends.
  2. Exponential backoff: wait 100 ms, then 200, then 400, because hammering a struggling service makes things worse.
  3. Jitter: add a random fraction to each delay, so a thousand clients that failed together do not all retry on the same millisecond.
  4. A retryable check: retry timeouts and 503s, but never retry a 400 or a ValidationError.

The delay and the randomness are injected so the test (and this lesson) stays deterministic.

Node.js Retry with exponential backoff and jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With jitter fixed at 0.5 the first two waits are 75 ms and 150 ms: 100 * 0.75 and 200 * 0.75. In real use random is Math.random, and the delays vary. The permanent error stopped after a single attempt, while the others used all they were allowed. Notice that the final error keeps the last real failure as its cause, so the log shows why it gave up.

Also bound the total time. Four attempts with backoff plus a 30-second timeout each can hold a user's request for minutes. Put a deadline on the whole operation with an AbortSignal.timeout() (see AbortController and AbortSignal) and check it before each retry.

A circuit breaker by hand

Retries help with brief hiccups. When a dependency is down, they do the opposite: every request waits through all its attempts, work piles up, memory grows and your service goes down together with the dependency. A circuit breaker stops calling a failing dependency for a while, failing fast instead.

It is a tiny state machine with three states:

  • closed: normal. Calls go through; failures are counted.
  • open: after too many failures, every call fails immediately with no network traffic, until a cooldown elapses.
  • half-open: after the cooldown, let one trial call through. If it succeeds, close the circuit; if it fails, open it again.
Node.js A circuit breaker with an injected clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at the real call counter: calls 4 and 5 never touched the dependency. That is the whole point. The service answers immediately with an error it can turn into a 503 or a cached fallback, and the struggling dependency gets breathing room to recover. Production libraries add a rolling window, a success threshold and metrics, but the core is what you just ran.

Combine the pieces in the right order: the retry wraps individual calls to ride out single failures, and the breaker sits around the whole retried operation (or around each attempt, depending on whether you want attempts to count) so repeated failure trips it. Do not retry a CircuitOpenError; make it non-retryable.

Dead letter queues for work you cannot finish

Background work (sending emails, processing uploads, consuming messages) has no user to show an error to. If a job fails permanently, you must not lose it silently and must not retry it forever. The standard answer is a dead letter queue (DLQ): after the maximum attempts, move the failed job, together with the error and the attempt count, to a separate place where a human or a repair job can inspect it, fix the cause and replay it.

Node.js A job queue with retries and a dead letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The healthy job finished, the flaky one succeeded on its second attempt, and the hopeless one landed in the dead letter list with its error and attempt count. In production the DLQ is a database table or a separate queue topic, and you alert when it is not empty. Two details separate a toy from a real system: jobs must be idempotent (a retry or replay must not send two emails), and a failure caused by a bug in your code should pause the whole consumer rather than empty the queue into the DLQ one message at a time.

A production checklist

  • Every error is either an AppError (expected, with a code and status) or treated as a bug.
  • One central handler logs, classifies and formats; routes just throw.
  • Unexpected errors show users a generic message and a request id, never internals.
  • uncaughtException and unhandledRejection log, flush and exit; a supervisor restarts.
  • Retries use a limit, exponential backoff, jitter, and only for retryable, idempotent operations.
  • A circuit breaker protects you from a dependency that is down.
  • Background jobs have a dead letter queue, and alerts on it.
  • Errors keep their cause chain from the failing driver up to the final message.
  • You test the failure paths: see Mocks, Spies and Fake Timers for scripting failures.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A request handler reads a property of undefined because of a missing null check. How should the system classify and treat it?

    A TypeError from a bug is a programmer error. Retrying cannot fix it, and its message can leak internals, so the user gets a generic message while the log keeps the details.
  2. 2. What is the recommended reaction to an uncaughtException event?

    After an uncaught exception the process state is unknown. A restart returns to a known-good state, and a supervisor automates it.
  3. 3. Why add jitter to a retry delay?

    Without jitter, many clients that failed together retry together and hit the recovering service in synchronized waves.
  4. 4. A circuit breaker is open. A new call arrives before the cooldown has passed. What happens?

    While open, the breaker fails fast. That protects both your service and the struggling dependency.
  5. 5. What belongs in a dead letter queue?

    A DLQ holds work that could not be completed, so it is neither lost nor retried forever, and a person can fix the cause and replay it.

Summary

  • Distinguish operational errors (the world misbehaved; handle them) from programmer errors (bugs; fix them and restart).
  • Give expected errors a shared base class with a stable code, an HTTP status and an expose flag.
  • Let errors propagate to one central handler that logs, classifies and formats; never leak internals for unexpected errors.
  • On uncaughtException or unhandledRejection, log, flush and exit non-zero; let a supervisor restart the process.
  • Retry only idempotent, retryable failures, with a limit, exponential backoff and jitter, and keep the last error as cause.
  • A circuit breaker (closed, open, half-open) fails fast while a dependency is down.
  • Use a dead letter queue for background jobs that cannot be completed, and alert on it.

Next: Graceful Shutdown and Health Checks