Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 133 ยท File System

File Streams

Process big files in Node.js without loading them into memory - createReadStream and createWriteStream, highWaterMark, byte ranges, reading lines with readline, copying with pipeline and tail-like reading.

readFile loads a whole file into memory before you can touch the first byte. That is fine for a 2 KB configuration file and a problem for a 4 GB log, a video, a database dump or a CSV export: the process uses gigabytes of RAM, the first result appears only after everything is read, and a few parallel requests can crash the server. Streams solve this by moving data in small pieces, called chunks. You read a chunk, process it, let it go, and read the next one. Memory use stays small and constant, however big the file is.

This lesson uses streams specifically for files. It teaches createReadStream and createWriteStream, the highWaterMark option, reading a byte range, reading text line by line with readline, copying files safely with pipeline, and a tail-like reader for the end of a file. The general theory of streams (the four types, backpressure, Transform) gets its own chapter, beginning with Streams Fundamentals. You can follow this lesson without it.

Writing a file with createWriteStream

createWriteStream(path) opens a file for writing and returns a Writable stream. You call write(chunk) as often as you like and end() when you are done. The 'finish' event (or finished() from stream/promises, or once(stream, 'finish')) tells you that all data has been handed to the operating system.

Node.js Writing a file in pieces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

By default createWriteStream truncates an existing file. Pass { flags: 'a' } to append instead (see the flags table in Reading and Writing Files). A write stream keeps one file descriptor open for its whole life, so appending a million log lines this way is far cheaper than a million appendFile calls.

There is one subtle thing about write(): it returns true or false. A false means "my internal buffer is full, please stop and wait for the 'drain' event". Ignoring that is how a fast producer fills the memory of the process when the destination is slow. We handle it below with pipeline, which does it for you.

Reading with createReadStream

createReadStream(path) returns a Readable stream. Use for await to consume it: the loop receives one chunk at a time, and the stream is paused while your loop body works, so you cannot be overrun.

Without an encoding the chunks are Buffers. The default chunk size is 64 KiB (65,536 bytes) for file streams. The highWaterMark option changes it.

Node.js Reading in chunks and counting them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The numbers are easy to verify: 1,000,000 bytes divided by 65,536 is 15.26, so 16 chunks (fifteen full ones and a shorter last one). With 16 KiB chunks you get 62; with a 1 MiB limit the whole file arrives in a single chunk.

Which size is best? Bigger chunks mean fewer system calls and fewer loop iterations, but more memory per chunk. The default is a good compromise; for plain file copies people sometimes raise it to 256 KiB or 1 MiB. Measure before tuning, and remember that the highWaterMark is a limit, not a promise: a chunk can be smaller.

Strings instead of buffers

Pass an encoding and chunks arrive as strings. Node.js uses a decoder that correctly handles a multi-byte character cut in half at a chunk boundary, which is the bug you get when you call toString() on each buffer yourself (see Buffers).

Node.js Encoding option and split characters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The naive version prints the replacement character ๏ฟฝ where a symbol was cut in half. The stream with an encoding yields clean strings (sometimes with a different number of characters per chunk, since a partial character is held back until the rest arrives).

Reading part of a file: start and end

start and end are byte offsets (both inclusive) and let you read a slice of a file without touching the rest. This is how HTTP Range requests, resumable downloads and video seeking work. It is also useful for reading a header or the end of a file.

Node.js Reading a byte range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Note that end is inclusive: start: 0, end: 4 gives five bytes. This differs from Buffer.subarray and Array.slice, where the end is exclusive, and is a classic off-by-one source. A range also splits a multi-byte character if it cuts through one, because offsets count bytes.

Reading line by line

Raw chunks do not respect line boundaries. A chunk may end in the middle of a line. The node:readline module reassembles lines for you: give it a stream and it yields complete lines as an async iterable.

Node.js Lines from a file with readline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The tiny 32-byte highWaterMark forces many cut lines, yet every line arrives whole. Memory stays proportional to the longest line, not to the file. The final line does not need a trailing newline. readline is explained more in Interactive CLIs with readline, and a hand-written splitter is the subject of Transform and Duplex Streams.

Use crlfDelay: Infinity whenever you read files that might come from Windows. Without it, a \r at the end of one chunk and a \n at the start of the next may be treated as two line breaks.

Copying with pipeline

To copy a file by streaming, connect a read stream to a write stream. Do not use the older readStream.pipe(writeStream) for this: it does not forward errors or close the other stream when one fails, which leaks file descriptors. Use pipeline from node:stream/promises. It connects the streams, applies backpressure (pausing the reader while the writer is busy), and destroys everything if any part fails.

Node.js Copying a file with pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

pipeline accepts any number of streams, so you can insert a transformation in the middle. A very common one is compression with the built-in zlib module, shown in Compression with zlib:

Node.js Gzip a file while copying it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

For a plain file copy without any transformation, copyFile (or cp) from the previous lesson is usually better: the operating system can copy the data without it ever entering your process, sometimes even by a copy-on-write trick that takes no extra disk space. Use streams when you need to transform the data on the way.

Backpressure in one picture

What happens when the reader is faster than the writer? Without control, data piles up in memory. The write stream has an internal buffer limited by its own highWaterMark (16 KiB by default for file writes). When write() returns false, the buffer is full. A producer that obeys waits for 'drain':

Node.js Respecting write() and drain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The loop never holds more than about 16 KiB of pending data in memory, instead of one million bytes. For one megabyte this hardly matters, but with gigabytes it is the difference between working and crashing. The exact number of pauses depends on the machine, so we only print whether it happened. Full details are in pipeline and Backpressure.

When your data comes from a generator or any async iterable, Readable.from(iterable) turns it into a stream, and pipeline(Readable.from(rows), createWriteStream(...)) writes it with correct backpressure without any manual drain code.

A tail-like reader

The Unix command tail -n 3 file prints the last lines of a file without reading everything. We can do the same with a range read: look at the file size, read only the final block of bytes, and take the last lines from that.

Node.js Reading the end of a big file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The function read a single 1,024-byte block of the 228,894-byte file, never the whole thing. A real implementation also needs to avoid cutting a multi-byte character at the start of the first block (decode after joining the buffers, as above, and drop a possibly broken first line when you did not reach the start of the file). To follow a growing log, combine this with fs.watch from the previous lesson, and remember to handle log rotation (the file is renamed and a new one is created).

Pitfalls and best practices

  • Errors. A read stream emits 'error' (for example ENOENT) on a missing file, not at the call site. Using for await or pipeline turns the event into a thrown error or rejected promise. An unhandled 'error' event crashes the process.
  • Close what you open. Streams close their file when they end or are destroyed. If you stop reading early (break in for await), the loop destroys the stream for you. With manual .on('data') code, call stream.destroy().
  • Do not mix reading modes on the same stream (data events, read() calls and for await).
  • Pick the right tool. Small file: readFile. Whole-file copy: copyFile. Transform or large data: streams. Line-oriented data: readline.
  • Memory sanity check. If a "streaming" job still uses huge memory, something is collecting every chunk (an array, a string += that you keep, a Promise.all over the lines).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use createReadStream instead of readFile for a 3 GB log file?

    readFile builds one buffer for the entire file. A stream delivers it chunk by chunk, so you never hold more than a few chunks.
  2. 2. What is the default chunk size (highWaterMark) of a file read stream?

    File read streams use 64 KiB (65,536 bytes). Write streams use 16 KiB.
  3. 3. createReadStream(path, { start: 2, end: 5 }) reads which bytes?

    Both offsets are inclusive byte positions, unlike slice and subarray where the end is exclusive.
  4. 4. Why prefer stream.pipeline over readStream.pipe(writeStream) for copying?

    pipe does not clean up the other stream when one fails, which leaks file descriptors. pipeline handles errors, cleanup and backpressure.
  5. 5. What does it mean when writeStream.write(chunk) returns false?

    The data is not lost, but you should pause until 'drain' to avoid unbounded buffering. pipeline does this automatically.

Summary

  • Streams move a file in chunks, keeping memory constant. Use them for large files or when you transform data on the way.
  • createReadStream yields 64 KiB Buffers by default; highWaterMark changes the size and encoding yields strings that never split characters.
  • start and end are inclusive byte offsets that read part of a file.
  • readline.createInterface({ input }) turns a stream into whole lines; set crlfDelay: Infinity.
  • createWriteStream keeps a file open; obey the return value of write() and the 'drain' event, or let pipeline do it.
  • pipeline from node:stream/promises copies and transforms safely, with error propagation and cleanup. Prefer copyFile for plain copies.
  • A tail-like reader reads only the last bytes of a file using stat and a range.

Next: Paths, Permissions and Safe File Handling