Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 133 ยท Streams

Transform and Duplex Streams

Build Node.js Transform and Duplex streams - transform() and flush(), a line splitter, a JSON-lines parser, objectMode on each side, PassThrough and pipelines of transforms.

Readables produce data and writables consume it. Between them there is room for a third kind of stream that does something with the data on the way through: compress it, decrypt it, split it into lines, parse it into objects, filter it. That is a Transform. It is the most useful stream type to write yourself, because it lets you build processing steps that snap together like pipe sections and never load the whole input into memory.

A Transform is a special case of a Duplex, a stream that is both readable and writable. This lesson explains the difference, shows how to write transforms with transform() and flush(), builds two real ones (a line splitter and a JSON-lines parser), and then connects them. Prerequisites: Readable Streams and Writable Streams.

Duplex: two independent streams in one object

A Duplex has a readable side and a writable side that do not have to be related. A TCP socket is the classic example: what you write goes to the other computer, and what you read comes from it. You implement both _read and _write yourself.

Node.js A Duplex with unrelated sides

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The two sides share an object, nothing more. Their buffers, events and end are separate; the stream is fully done only when both sides finish. In practice you rarely write a pure Duplex. You get them from sockets, or you build a transform.

Transform: output computed from input

A Transform is a Duplex where the readable side is the result of the writable side. Instead of _read and _write you implement one method:

transform(chunk, encoding, callback): receive a chunk, call this.push(result) any number of times (zero, one, or many), then call callback() to say you are ready for the next chunk. A shortcut: callback(null, result) pushes one result and calls back at once.

Node.js The smallest useful Transform

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Chunks in, chunks out. pipeline (covered fully in pipeline and Backpressure) connects the steps and cleans up after errors; the last step here is an async function that consumes the output.

A transform can emit zero, one or many chunks

The count of output chunks does not have to match the input. A filter pushes nothing for items it rejects. A splitter pushes several. A compressor may push nothing for a while and then a big block. This makes transforms the right tool for line splitting, filtering and batching.

Transform errors

Pass an error to the callback, callback(new Error('...')), or throw inside an async transform. The stream is destroyed, and pipeline rejects and tears down all the other streams.

flush: the last word

Many transforms keep state between chunks, and when the input ends, that state may contain unfinished work. flush(callback) runs after the last chunk has been transformed and before the readable side ends. Use it to push whatever is left.

The classic case is splitting text into lines. A chunk can end in the middle of a line, so the transform must remember the partial line until the rest arrives, and emit it at the end if the file does not finish with a newline.

Node.js A line splitter with state and flush

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Study the pieces. this.leftover carries the unfinished line across chunks. lines.pop() removes it from the complete lines. _flush emits the remainder, which is the line that did not end with a newline. A TextDecoder with { stream: true } handles characters split across chunks, the problem from Streams Fundamentals. Node also has readline for this exact job on files and stdin, but writing it yourself shows the mechanics every parser shares.

readableObjectMode and writableObjectMode can be set separately. Here the input side is a byte stream and the output side carries JavaScript strings, one per line. The shortcut objectMode: true turns on both sides.

Parsing JSON Lines

Many tools exchange data as JSON Lines (also called NDJSON): one JSON value per line. It is ideal for streaming because each line is parsed on its own. Building a parser from the splitter gives you a stream of objects.

Node.js JSON Lines parser in objectMode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first input parses into three records (the blank line is skipped). In the second, line 2 is broken: the parser fails with the line number in the message and pipeline rejects, destroying every stream in the chain. The run reports 0 records because the error happens in the same burst of work that would have delivered record 1, so the chain is torn down first. Nothing keeps running after an error. Notice also how little memory this needs: whatever the input size, only one line at a time is being parsed.

PassThrough: a transform that does nothing

PassThrough is a Transform that passes every chunk through unchanged. That sounds useless, but it is a handy adapter: a connection point where you need a stream but only have a place to plug one in. Typical uses: tapping into a stream to observe it, and giving a library a writable stream whose output you read yourself.

Node.js PassThrough as a tap and a connector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The meter sees 'abc' and 'defg' pass by (seven bytes), then the data continues on to the uppercase step. Attaching a 'data' listener to a PassThrough that is also piped onward is acceptable here because the meter only observes; for anything more elaborate, write a proper Transform whose transform method does the counting and then forwards the chunk.

Composing a pipeline of transforms

The power of transforms is that they stack. Each does one thing; the pipeline wires them. Below, a stream of text lines flows through three steps: a numbering step, a filter that drops comment lines, and a formatter. The result is an ordinary object-mode stream you could send to a file or a socket instead.

Node.js Three transforms in a row

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Five lines went in; two of them were comments that produced no output, and three formatted lines came out. Each step is independent and testable on its own, and swapping one (say, reading from a file instead of an array) does not touch the others. Node also offers stream.compose() to wrap a chain like this into one reusable stream; see Stream Patterns and Recipes.

Pitfalls

  • Forgetting the callback. If transform never calls callback, the stream stalls without an error. Every code path, including early returns, must call it exactly once.
  • Calling it twice. That raises ERR_MULTIPLE_CALLBACK.
  • Losing state at the end. Anything you keep between chunks needs a flush to be emitted.
  • Forgetting the object-mode flags. Pushing an object from a byte-mode readable side throws. Set readableObjectMode when you output objects and writableObjectMode when you receive them.
  • Writing a transform where an async generator would do. Async generators passed to pipeline act as transforms with far less ceremony (see Stream Patterns and Recipes). Write a class when you need flush semantics, reuse across several pipelines, or the stream interface itself.
  • Heavy synchronous work. A transform that spends 50 ms per chunk on the CPU blocks the whole event loop. Move such work to a worker thread.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What distinguishes a Transform from a plain Duplex?

    A Duplex has two independent sides. A Transform wires them together: you implement transform(), and what you push becomes the output.
  2. 2. A transform needs to emit one final piece of buffered text after the input ends. Where does it do that?

    flush(callback) is the hook for leftover state such as a last line without a trailing newline.
  3. 3. How many chunks can one call of transform() push?

    Filters push zero, splitters push many. Input and output chunk counts do not have to match.
  4. 4. A parser receives bytes and outputs JavaScript objects. Which options fit?

    The output (readable) side carries objects, so readableObjectMode is true while the input side stays in byte mode.
  5. 5. What happens if transform() never calls its callback?

    The callback is how the stream knows it may send the next chunk. Forgetting it is a silent hang.

Summary

  • A Duplex has independent readable and writable sides; a Transform ties the output to the input and is implemented with transform(chunk, encoding, callback).
  • A transform can push zero, one or many chunks per input chunk, and must call its callback exactly once per call.
  • flush(callback) runs after the last input chunk; use it to emit leftover state such as a final unterminated line.
  • readableObjectMode and writableObjectMode configure each side separately; objectMode: true sets both.
  • A line splitter plus a per-line parser gives a streaming JSON Lines reader that uses constant memory and reports failures with line numbers.
  • PassThrough forwards data unchanged and is useful as a tap, a mailbox or a connector.
  • Stack small, single-purpose transforms in pipeline() instead of writing one big step.

Next: pipeline and Backpressure