Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 133 ยท Streams

Stream Patterns and Recipes

Practical Node.js stream recipes - async generators as transforms, stream.compose, batching, throttling, merging, splitting and processing a huge CSV with flat memory.

You now know the four stream types and how pipeline and backpressure keep them safe. This lesson is a cookbook. It collects the patterns that come up again and again in real programs: writing a transform as a plain async generator, gluing several steps into one reusable piece, grouping items into batches, slowing a source down, merging and splitting flows, and finally chewing through a very large CSV file without ever holding it in memory.

None of these patterns needs a package. They are small, they compose, and once you have seen the shape of each one you will recognize it in other people's code. Every example runs on our server, uses only the standard library, and prints the same thing on every run.

Async generators as transforms

The shortest way to write a transform is not a Transform class at all. stream.pipeline accepts an async generator function as a step. It receives the previous stream as an async iterable, and whatever it yields becomes the data for the next step. Backpressure still works: a generator only runs ahead when the consumer pulls.

Node.js A pipeline made of generators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Each generator is a small, testable function: it takes an async iterable and returns an async iterable. A generator can also yield zero, one, or many values per input, so it can filter (as dropShort does), map, or expand. Flushing leftover state at the end is easy too: put the code after the loop. What a generator cannot do is run on a timer of its own (for example "flush a batch after one second"), and for that the full Transform class is still the right tool.

If the last step of pipeline is an async function (not a generator) it acts as the destination: it receives the previous step as an async iterable and its returned promise decides when the pipeline ends. That removes the need for a Writable in many scripts.

Node.js An async function as the final step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The value returned by the last function is what await pipeline(...) resolves to, which makes this a neat way to reduce a stream to one answer.

Reusable pieces with stream.compose

Sometimes you want to name a sequence of steps, such as "decode, split into lines, parse JSON", and reuse it in several pipelines. stream.compose(a, b, c) connects its arguments and returns one duplex stream: you write into the first step and read from the last. Errors and destroy() calls travel through the whole chain.

Node.js Composing steps into one stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at the input: the second object is cut in the middle ({"id" and :2} arrive in different chunks). splitLines keeps the unfinished text in rest until the newline arrives, so the parser only ever sees whole lines. This "carry the leftover" trick is the heart of every text-splitting stream.

stream.compose was added in Node.js 16.9 and is still marked experimental in the documentation. It has been stable in practice for years, but if you want zero risk, write a small function that returns an async generator instead: async function* jsonLines(source) { yield* parseJson(splitLines(source)); }. Generators nest by calling each other.

Batching

Writing one database row, one HTTP request or one log line per item is often painfully slow. Collecting items into groups of N and handling the group at once is called batching, and an async generator expresses it in a few lines. Remember the leftover: the last group is usually smaller.

Node.js Batching items into groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Eleven items with a batch size of 4 give three calls: 4, 4 and 3. A common refinement is a time limit as well as a size limit ("flush every 500 rows or every second, whichever comes first"), which needs a timer next to the loop and is easier with a class-based Transform and its flush method.

Throttling and pacing

A fast source can overwhelm a slow API. Because a generator only advances when it is pulled, putting a delay inside a generator slows the whole pipeline down: the upstream readable simply waits (that is backpressure doing its job). timers/promises gives you an awaitable delay.

Node.js Limiting how fast items flow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

We print only a yes/no about the elapsed time, because exact durations vary between runs. For real rate limits (for example "10 requests per second") you will usually combine pacing with the concurrency control tools later in this track.

Merging and splitting

Merging several sources into one is a loop that pulls from whichever source is ready. The simplest deterministic merge reads the sources in order, one after the other. When you need "first come, first served" you race the next value of each source with Promise.race; the resulting order depends on timing, so tests should sort or compare sets.

Node.js Concatenating and interleaving sources

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

interleave alternates a 1 b 2 c 3, then drains whatever is left (d). Notice it waits for each source in turn, so one slow source slows the others: it is deterministic, not fast. A truly "as they arrive" merge needs Promise.race over pending next() calls and careful bookkeeping; if you need it in production, consider a well-tested library.

Splitting one stream into several (a fan-out) means every consumer should receive every item, or each item should go to exactly one consumer. A simple, safe version is to route items by a rule in the final step.

Node.js Routing items to different destinations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Avoid source.pipe(a); source.pipe(b); for fan-out. The source then runs at the speed of the fastest consumer (a slow one buffers without limit) and an error in one branch is not propagated to the other. If both branches really must see the data, push to both writables from one loop, await-ing a drain event whenever write() returns false.

Processing a large CSV with flat memory

Now we combine the patterns: read bytes, turn them into lines, parse each line, aggregate, and count how many rows were alive at the same time. A real run would read createReadStream('sales.csv'). Our example generates the same shape of data in the program, 50,000 rows delivered in 64-byte-ish chunks that cut rows in half, so the carry-over logic is really exercised.

Node.js Aggregating 50,000 rows with a stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The important output is the last line: only one row was alive at a time, even though 50,000 went through, because each generator hands over a single line and waits to be pulled again. Memory use depends on the size of one chunk, not on the size of the file. That is the entire promise of streams, and you proved it with a counter instead of a guess. (Never print real memory numbers in tests; they differ between machines. A counter of items in flight is stable and meaningful.)

Pitfalls professionals avoid

  • Lines split across chunks. Always carry the unfinished tail (rest) to the next chunk. Forgetting this corrupts one row per chunk boundary, which is the hardest kind of bug: rare and data-dependent.
  • Quoted CSV fields. A comma or newline inside "quotes" breaks the naive split. For real CSV, use a state-machine parser, and remember that a quoted field can contain a newline, so "split on newline first" is not safe for such files.
  • Multi-byte characters. If your chunks are Buffers, a UTF-8 character can be cut in half. Call readable.setEncoding('utf8') or decode with TextDecoder using { stream: true }; see Text Encoding.
  • Swallowed errors. Use pipeline, not chained .pipe() calls, so an error in any step destroys every other step.
  • Unbounded buffers inside a step. A generator that collects everything before yielding has turned your stream back into a big array.

When to use and when to avoid

Reach for these patterns when data is large, unbounded, or arrives over time: files, sockets, exports, logs, API pagination. Avoid them when the whole thing is small (a 2 KB JSON config does not need a pipeline) or when the code must be random-access. Streams add moving parts; use them where they remove a real memory or latency problem.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What can an async generator function do when it is used as a step in stream.pipeline?

    Each generator step takes an async iterable (the previous stream) and yields values that become the input of the next step, with backpressure handled for you.
  2. 2. Why does a line-splitting step keep a rest variable?

    Chunks end at arbitrary byte positions, so the last piece may be half a line. It is kept and prepended to the next chunk.
  3. 3. What does stream.compose(a, b) return?

    compose connects its arguments and gives you one duplex stream; errors and destroy() propagate through all the steps.
  4. 4. Why is source.pipe(a); source.pipe(b); a poor way to fan out to two consumers?

    With two pipes the slow branch buffers without a bound and the branches are not tied together for error handling.
  5. 5. How do you slow down a whole pipeline from inside one step?

    A step that awaits a delay does not pull the next item until it is ready, so upstream waits too. This is backpressure.

Summary

  • An async generator is the shortest way to write a transform: take an async iterable in, yield values out.
  • The last step of pipeline can be a plain async function, and its return value is what await pipeline(...) gives you.
  • stream.compose joins steps into one reusable duplex stream.
  • Batch items to cut per-item overhead, and always flush the final, smaller batch.
  • Delaying inside a generator throttles the whole pipeline thanks to backpressure.
  • Carry unfinished lines (and unfinished UTF-8 characters) from one chunk to the next.
  • Prove flat memory with counters of items in flight, not with memory numbers.

Next: Callbacks and Error-First Style