Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 133 ยท Streams

Readable Streams

Create and control Node.js readable streams - Readable.from, subclassing with _read and push, objectMode, highWaterMark, the readable event, destroy and error handling.

A readable stream is a source of data that you consume. Most of the time you receive readables from Node: a file, an HTTP request, a socket. But sooner or later you need to create one, for example to expose rows from a database as a stream, to generate test data, or to wrap a callback-based library. This lesson shows every way to create a readable, explains the pull model that drives it (_read and push), and covers the options and failure modes that matter in real programs.

If the words "flowing", "paused" and "chunk" are new, read Streams Fundamentals first.

The easy way: Readable.from

Readable.from() wraps any iterable or async iterable. For most custom sources, an async generator is the shortest and clearest code, and it handles waiting for you.

Node.js An async generator as a readable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice that Readable.from defaults to object mode: each yielded value is one chunk, whatever it is. Pass { objectMode: false } when you yield Buffers or strings and want a byte stream.

The pull model: _read and push

When you need full control, subclass Readable and implement _read(size). Two rules define everything:

  1. Node calls _read() when the stream wants more data. You respond by calling this.push(chunk) zero or more times.
  2. Calling this.push(null) signals the end of the stream.

The important word is wants. A readable only asks for data when its internal buffer has room and someone is consuming. This is how a slow consumer slows down a fast producer: if nobody reads, _read is not called again. The demo prints a message each time Node asks.

Node.js A counter that is read on demand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output carefully. _read runs once, the consumer receives a number, then Node calls _read again for the next one, and the last call pushes null. Nothing is produced before a consumer starts asking. This laziness is the heart of streaming.

_read may push several chunks per call, push asynchronously (call push later, from a callback), or push nothing yet. If you push nothing and return, Node will not call _read again until you do push something. So an async source must call push when its data arrives:

NODE
// Sketch: wrapping an event-based source (read-only example)
class FeedReader extends Readable {
  constructor(feed) {
    super({ objectMode: true });
    feed.on('item', (item) => {
      if (!this.push(item)) {
        feed.pause();          // buffer is full: stop the source
      }
    });
    feed.on('end', () => this.push(null));
    this._feed = feed;
  }
  _read() {
    this._feed.resume();       // consumer wants more: restart the source
  }
}

push() returns false when the internal buffer is full. That is the readable's way of telling you to stop producing until _read is called again. Ignoring it is how memory balloons.

highWaterMark: how much to buffer

Every stream has a highWaterMark, the buffer size at which it starts saying "enough for now". For byte streams it counts bytes (the default is 16,384 bytes, 16 KiB, for a plain Readable; file streams choose a larger value for themselves). In object mode it counts objects (default 16).

The mark is a threshold, not a hard limit: push still accepts data over it, it just returns false. The following demo shows the buffer filling and push giving the warning.

Node.js Watching push() return false

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

You can see push return false at the third item and keep accepting items afterward. Nothing was lost; the stream simply asked its producer to slow down.

Passing read() {} in the options is the shortcut for a stream you will feed from outside. It avoids creating a subclass.

Strings, buffers and encodings

In byte mode, push accepts a Buffer, a Uint8Array or a string. A string is converted to a Buffer using the encoding option (UTF-8 by default). Consumers see Buffers unless they call setEncoding. Mixing numbers or objects into a byte-mode stream throws ERR_INVALID_ARG_TYPE, which is the number-one beginner error when someone forgets objectMode.

Node.js Pushing the wrong type (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The real message is ERR_INVALID_ARG_TYPE: the chunk argument must be a string, Buffer, TypedArray or DataView. The fix is to add objectMode: true (then every chunk is a value) or to serialize the object yourself, for instance with JSON.stringify(obj) + '\n'.

Reading modes in practice

You can consume a custom readable any way described in the previous lesson. Two details are specific to creating readables.

read(size) (paused mode) returns exactly size bytes if that many are buffered, otherwise null, unless the stream has ended, in which case it returns whatever remains. This is how binary parsers pull fixed-size headers.

Node.js Taking exact amounts with read(size)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Async iteration vs events. for await is the least error-prone. Events give you finer control but mean handling 'error', 'end' and backpressure yourself. When in doubt, iterate.

destroy() and errors

Streams end in one of two ways: gracefully (push(null), then 'end') or by being destroyed. stream.destroy() releases resources immediately and emits 'close'. stream.destroy(err) additionally emits 'error' with your error first. After destruction, no more data is delivered.

Implement _destroy(err, callback) to clean up your own resources: close a file, cancel a timer, unsubscribe from the feed. Always call the callback.

Node.js Cleanup with _destroy and an error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The break destroyed the stream with an AbortError ("The operation was aborted"), which for await swallows for you. With for await, the error becomes an exception at the loop, and break destroys the stream so _destroy runs and the source can clean up. With events you would listen for 'error' yourself; without a listener, the error is thrown as uncaught.

An 'error' event on a stream is terminal. After it, the stream is destroyed and will not emit 'end'. Do not try to keep reading. Create a new stream if you want to retry.

Designing a good readable

  • Be lazy. Do the work in _read, not in the constructor. Opening a file or database cursor when nobody reads wastes resources and can leak them.
  • Respect push() returning false. Stop producing until _read is called again.
  • Always end. Forgetting push(null) leaves consumers waiting forever.
  • Prefer an async generator. If you do not need to manage buffering by hand, Readable.from(asyncGenerator()) is shorter and has fewer ways to go wrong.
  • Pick the right mode. Bytes for files and network protocols, objects for parsed records.
  • Do not mix. One consumption style per stream.

Next you will see the other side of the pipe, Writable Streams, and later how to connect the two safely with pipeline and Backpressure.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. When does Node call a readable's _read() method?

    Readables are pull-driven: _read is called when more data is wanted, so a slow consumer slows the producer.
  2. 2. What does stream.push(null) do?

    push(null) marks the end of the data. After the buffer drains, consumers see the 'end' event.
  3. 3. What does it mean when push() returns false?

    The chunk is still buffered. false is only a hint to stop producing for now.
  4. 4. In object mode, what does highWaterMark count?

    In object mode each chunk is one object, so the default of 16 means about 16 objects.
  5. 5. You push an object into a stream created without objectMode. What happens?

    Byte streams only accept strings, Buffers, TypedArrays and DataViews. Use objectMode: true or serialize the object first.

Summary

  • Readable.from(iterable) is the shortest way to make a readable; it defaults to object mode.
  • Subclass Readable and implement _read() for full control: call push(chunk) to supply data and push(null) to end.
  • Readables are pull-driven and lazy; push() returning false means the buffer passed highWaterMark and you should pause.
  • highWaterMark counts bytes in byte mode and objects in object mode; it is a threshold, not a hard limit.
  • Byte streams accept strings, Buffers and TypedArrays only; use objectMode for other values.
  • destroy(err) emits 'error' then 'close'; implement _destroy to clean up, and break in for await destroys the stream.

Next: Writable Streams