Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 133 ยท Streams

Streams Fundamentals

Learn Node.js streams from scratch - why they exist, the four stream types, chunks, flowing and paused modes, events, encodings, Readable.from and async iteration.

Imagine you must copy a 40 GB log file. You could read all of it into memory, then write it out. That needs 40 GB of RAM, nothing happens until the whole file is loaded, and the process is killed long before it finishes. Or you could read a small piece, write that piece, and repeat. Memory stays tiny, the first bytes arrive at the destination right away, and the file can be as large as the disk allows. The second approach is what a stream is: data that you handle piece by piece, as it arrives, instead of all at once.

Streams are one of the oldest and most important ideas in Node.js. Files, network sockets, HTTP requests and responses, compression, process.stdin and process.stdout and child-process pipes are all streams. Learn the model once and all of them make sense. This lesson covers the vocabulary, the four types, how data flows and how to consume it. The next lessons go deep: Readable Streams, Writable Streams, Transform and Duplex Streams and pipeline and Backpressure.

Why streams: memory and time

Streams solve two different problems.

Memory. Processing a chunk at a time means memory use depends on the chunk size, not on the data size. Ten megabytes and ten terabytes cost about the same.

Time. Work can overlap. While the next chunk is still being read from the disk, the previous chunk is already being compressed, and the one before that is being sent over the network. With "read everything, then process, then send", those steps run one after another.

The demo below makes the memory argument concrete without relying on real memory numbers. A generator produces 1,000 pieces of 1,000 bytes each, which is one megabyte in total. We count the biggest piece that was ever held at one time.

Node.js One megabyte, one chunk at a time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The loop saw a megabyte but never held more than the piece it was working on. (A stream may merge several small pieces into one bigger chunk, so the chunk count you see for a given source is not always the number of writes that produced it. Treat chunk boundaries as an implementation detail.)

A simple rule of thumb: if the data could be large, unbounded or slow, use a stream. If it is a small config file, just read it.

The four stream types

Node has four base classes, all in node:stream. Every concrete stream is one of them.

Type Direction Examples
Readable You read data out of it fs.createReadStream, HTTP request on the server, process.stdin
Writable You write data into it fs.createWriteStream, HTTP response on the server, process.stdout
Duplex Both, independently A TCP socket: what you send and what you receive are separate flows
Transform A duplex whose output is computed from its input zlib.createGzip(), crypto.createHash() as a stream

Think of plumbing. A readable is a tap, a writable is a sink, a transform is a filter fitted in between, and a duplex is a two-way pipe such as a telephone line. You build programs by connecting them: tap, filter, filter, sink.

All streams are also event emitters. That is how they tell you "here is data", "there is no more data" and "something went wrong".

Chunks, strings and objects

Streams move chunks. By default a chunk is a Buffer (raw bytes, see Buffers). Two settings change that:

  • setEncoding('utf8') on a readable makes chunks arrive as strings. Node decodes for you, and does it correctly when a multi-byte character is split across two chunks.
  • objectMode: true lets a stream carry any JavaScript value (objects, numbers) as a chunk instead of bytes. Then one chunk is exactly one value and nothing is merged or split. The Readable Streams lesson covers it in detail.

The reason setEncoding matters is that bytes and characters are not the same thing. The euro sign is three bytes in UTF-8, and a chunk boundary can land in the middle of them.

Node.js A character split across two chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The naive version produced three replacement characters (one for the lone first byte, two for the leftover bytes) because each part of the euro sign is invalid UTF-8 on its own. With setEncoding, the stream holds the incomplete bytes back until the rest arrives. The same trap exists whenever you concatenate chunk.toString() yourself, so prefer setEncoding, or collect Buffers first with Buffer.concat. See Text Encoding for the bigger picture.

Flowing and paused: how a readable delivers data

A readable stream starts out paused: it has data buffered inside but does not hand it out until you ask. There are three equivalent ways to start consuming it, and which you pick decides the mode.

  1. Attach a 'data' listener. The stream switches to flowing mode and emits 'data' for every chunk as fast as it can.
  2. Use pipe() or pipeline() to connect it to a writable, which does the flowing for you with flow control.
  3. Pull with read() after a 'readable' event (paused mode), or loop with for await.

You should pick one style per stream and stay with it. Mixing 'data' listeners and 'readable' listeners on the same stream leads to confusing behavior.

Flowing mode with events

Four events cover almost everything:

  • 'data' : a chunk is available.
  • 'end' : there is no more data to read (everything was consumed).
  • 'close' : the stream and its underlying resource (file descriptor, socket) are released.
  • 'error' : something failed. As with any emitter, an error event with no listener throws.
Node.js The readable lifecycle in events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice the first line prints before any data. Attaching a listener starts the flow, but the chunks are delivered on later turns of the event loop, never synchronously inside on().

Paused mode with read()

In paused mode you decide when to take data. Listen for 'readable', which fires when there is something to read (and again at the end), then call read() until it returns null.

Node.js Pulling data with read()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

'readable' can fire more than once (the repeated last line above is the extra firing near the end of the stream), and read() returns null when the internal buffer is empty. That does not mean the stream is finished; it means "come back after the next 'readable'". The 'end' event is the real finish line. Paused mode is lower-level and rarely needed in application code; it matters when you write parsers that must take exactly N bytes.

Switching between the modes: calling pause() stops a flowing stream, resume() restarts it, and readable.readableFlowing tells you the current state (null means nobody has chosen yet, true flowing, false paused).

The modern way: async iteration

Streams are async iterables. A for await loop handles flowing, ending and errors with ordinary code, and you get automatic flow control: the loop only asks for the next chunk when its body is finished.

Node.js for await over a stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

This is the style to reach for first. It reads top to bottom, try/catch works, and break cleans up the stream by destroying it. More on async iterators in Async Iterators.

Readable.from: streams from anything iterable

Readable.from(iterable) turns an array, a generator or an async generator into a readable stream. It is the quickest way to test stream code and to adapt your own data sources. One detail worth knowing: a bare string or Buffer is treated as a single chunk, not iterated character by character. To send several strings, put them in an array.

Reading all of a stream

Sometimes you really do want everything, for example a small HTTP response body. Node ships helpers in node:stream/consumers, so you do not need to write the loop yourself.

Node.js Collecting a whole stream with consumers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Use these only for data you know is small. They defeat the purpose of streaming because they buffer everything.

Common mistakes

  • Forgetting the 'error' listener. An unhandled 'error' event throws and can crash the process. Use pipeline() (see pipeline and Backpressure) so errors are always handled.
  • Treating a chunk as a complete message. A chunk is an arbitrary slice of bytes. A line, a JSON document or a UTF-8 character can start in one chunk and end in the next. Parse with a stateful transform instead (see Transform and Duplex Streams).
  • Mixing consumption styles. Choose 'data', 'readable' or for await for one stream, not several.
  • Buffering a "stream" into a string. Concatenating every chunk into one variable brings back the memory problem you came to avoid.
  • Ignoring backpressure when writing. Producing data faster than the destination accepts it fills memory. The Writable Streams lesson shows how write() signals this.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main memory advantage of a stream over reading a whole file with readFile?

    A stream processes one chunk at a time, so a huge file needs about as much memory as a tiny one.
  2. 2. Which stream type is a TCP socket?

    A socket has two independent flows (incoming and outgoing data), which is exactly what a Duplex stream models. A Transform is a duplex whose output depends on its input.
  3. 3. What makes a paused readable stream switch to flowing mode?

    Adding a 'data' listener, calling pipe() or calling resume() puts the stream in flowing mode.
  4. 4. Why is concatenating chunk.toString() yourself risky for text?

    Chunk boundaries can fall inside a multi-byte UTF-8 character. setEncoding() keeps the partial bytes until the rest arrives.
  5. 5. What does read() returning null mean in paused mode?

    null only means nothing is buffered at the moment. The 'end' event signals that no more data will ever come.

Summary

  • A stream handles data piece by piece, which keeps memory flat and lets work overlap in time.
  • The four types are Readable, Writable, Duplex and Transform; all are event emitters.
  • Chunks are Buffers by default; setEncoding gives strings (and decodes correctly across chunk boundaries) and objectMode carries arbitrary values.
  • Readables are paused until you attach a 'data' listener, use pipe/pipeline, call read() after 'readable', or iterate with for await.
  • Prefer for await and pipeline; use node:stream/consumers only for small data.
  • Chunk boundaries mean nothing: never assume a chunk is a whole line, message or character.

Next: Readable Streams