Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 133 ยท Streams

pipeline and Backpressure

Connect Node.js streams safely with stream.pipeline - error propagation, cleanup, backpressure explained with counters, the pitfalls of pipe(), and cancelling with AbortSignal.

You now know how to build readables, writables and transforms. The last piece is connecting them so that data flows at the right speed and failures are handled properly. Two ideas do all the work. Backpressure makes a slow consumer slow down a fast producer, so memory stays flat. pipeline() wires streams together, passes along errors and destroys every stream when anything goes wrong, so nothing leaks.

Together they are the difference between a script that works on a sample file and a service that survives a 20 GB upload, a client that disconnects midway and a disk that fills up. Earlier lessons: Streams Fundamentals, Writable Streams and Transform and Duplex Streams.

What backpressure is

Picture a fast tap filling a bucket with a small hole. If the tap never slows down, the bucket overflows. Streams have the same problem: a readable can often produce data faster than a writable can consume it (reading from a fast disk, writing to a slow network). The extra data has to be stored somewhere, and that somewhere is your process's memory.

Node's solution is a signal that travels backward through the chain:

  1. The writable's write() returns false when its buffer is over highWaterMark.
  2. The producer pauses until the writable emits 'drain'.
  3. Then it continues.

The mechanism was shown in Writable Streams. The demo below measures what it saves, with counters instead of memory numbers. A producer writes 1,000 chunks to a slow consumer in two ways: ignoring the return value, and honoring it.

Node.js Backpressure measured with counters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Without backpressure, all 1,000 chunks were queued at once. With it, the queue never held more than 9, just under the high water mark of 10. Now imagine the chunks are 64 KiB each and there are millions of them: the first version runs out of memory, the second does not notice.

Backpressure is not an extra feature you turn on. It is built into pipe() and pipeline(), and into for await loops over readables (the loop does not ask for more until the body finishes). You only have to do it by hand when you call write() yourself.

stream.pipeline

pipeline(source, ...transforms, destination) connects streams in order, applies backpressure between each pair and gives you one place to handle the outcome. It exists in two flavors:

NODE
// callback style
import { pipeline } from 'node:stream';
pipeline(source, transform, destination, (err) => {
  if (err) console.error('pipeline failed', err);
  else console.log('pipeline done');
});

// promise style (preferred)
import { pipeline } from 'node:stream/promises';
await pipeline(source, transform, destination);

Use the promise version in modern code. It resolves when the last stream has finished and rejects with the first error from any stream.

The steps do not have to be stream objects. Any of these are accepted: a readable, an iterable or async iterable as the first step; async generator functions in the middle (each receives the previous step's output as an async iterable and yields its own); and a writable or an async function (consuming the source) as the last step.

Node.js A promise pipeline with generator steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Generators are the lightest way to write a transform. Reach for a Transform class when you need flush semantics or reuse (see Transform and Duplex Streams).

Error propagation and cleanup

This is where pipeline earns its place. If any stream fails, pipeline destroys all the others and rejects with the error. Open files are closed, sockets released, and nothing is left running or leaking.

Node.js One failure destroys the whole chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Every stream, including the ones that did nothing wrong, was destroyed. Nobody has to remember to call destroy(). Compare that to the old way.

The problem with pipe()

readable.pipe(writable) predates pipeline and is still around, so you will meet it in older code and tutorials. It handles backpressure, but it has serious gaps:

  • It does not forward errors. An error in the destination or in the source does not reach the other one, and pipe() returns the destination, so a.pipe(b).pipe(c) only lets you catch errors on c.
  • When the destination fails, it does not destroy the source. The source keeps its resources, a file descriptor, for instance, until the process exits.
  • When the source fails, the destination is left open and waiting.
Node.js pipe() leaves the source open

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With pipe(), the sink's own error handler ran, yet the source was still alive (a leak in a real program). With pipeline() the source was destroyed automatically. Use pipeline() for every connection you make. The only reasonable use of pipe() today is a quick demonstration.

Backpressure inside a pipeline

You do not write drain handling in a pipeline. Each link works like this: when the writable side of step N is full, step N-1 is paused; its buffer then fills, and so on back to the original source. The consequence is that memory use of the whole chain is bounded by the sum of the highWaterMark values, no matter how large the data is.

The following example uses a slow final stage and a fast source that counts how many chunks it has produced. If backpressure works, the source cannot run far ahead of the sink.

Node.js The source is held back by the sink

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The source never got more than 6 chunks ahead of the sink (about the two high water marks of 4, give or take the chunk in flight), although it could have produced all 200 immediately. That bounded lead is backpressure at work. Without it, the gap would grow to nearly 200.

Cancelling a pipeline with AbortSignal

Both pipeline flavors accept { signal } as the last argument. When the signal aborts, pipeline destroys all streams and rejects with an AbortError. This is how you implement timeouts and "user closed the tab" cancellation. See Abort Signals for the general mechanism.

Node.js Cancel a pipeline after a deadline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

AbortSignal.timeout(5000) gives you a ready-made signal that fires after five seconds, so a stuck download cannot hang your program forever. In a server, pass a signal that you abort when the client connection closes.

Practical guidelines

  • Always use pipeline (promise version) to connect streams. Never chain .pipe() in new code.
  • Handle the rejection. Wrap await pipeline(...) in try/catch, and in HTTP handlers send an error response (if headers are not sent yet) or destroy the socket.
  • Do not add your own 'error' listeners on streams you hand to pipeline; it manages them.
  • Do not buffer in the middle. A transform that collects everything before emitting turns your stream back into readFile.
  • Size buffers deliberately. A larger highWaterMark can improve throughput for files, a smaller one saves memory for thousands of connections. Measure before changing the defaults.
  • Hand-written loops need drain. If you call write() in a loop yourself, check its return value every time.
  • Let finished() or pipeline tell you when it's done, not a lone 'finish' listener.

You will apply all of this to real files in File Streams, and to compression in Zlib.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does backpressure accomplish in a stream chain?

    When a writable's buffer is full, the producer is paused until it drains, which bounds the memory the chain needs.
  2. 2. A transform in the middle of a pipeline() call fails. What happens to the other streams?

    pipeline tears down every stream in the chain so no files or sockets leak, then rejects with the first error.
  3. 3. What is a major weakness of a.pipe(b)?

    pipe() does handle backpressure, but errors are not propagated and the source can be left open, which leaks resources.
  4. 4. Which of these can be a middle step in stream/promises pipeline()?

    Async generator functions take the upstream async iterable and yield transformed values, so they act as transforms.
  5. 5. How do you cancel a running pipeline from outside?

    When the signal aborts, pipeline destroys all streams and rejects with an AbortError.

Summary

  • Backpressure is the signal that travels backward through a chain: write() returns false, the producer waits for 'drain', and memory use stays bounded.
  • pipeline() from node:stream/promises connects streams, applies backpressure automatically, and settles when the last stream finishes or the first one fails.
  • On failure pipeline destroys every stream in the chain, so files and sockets are always released.
  • Async generator functions can be middle steps and iterables can be sources, which makes small transforms easy to write.
  • pipe() does not propagate errors or destroy the source; avoid it in new code.
  • Pass { signal } to cancel a pipeline for timeouts or disconnects.

Next: Web Streams in Node.js