Skip to content
Tutorials Center
My learning

Topics

Lesson 109 of 133 ยท Production Node.js

Processing Large Data Efficiently

Process big files and datasets in Node.js without running out of memory - streaming vs buffering, backpressure, batching database writes, worker pools and resumable jobs.

Everything works on your laptop with a 50 KB test file. Then production hands you a 6 GB export, and the process dies with JavaScript heap out of memory, or worse, it limps along while every other request on the server slows down. Handling large data in Node.js is mostly one idea applied over and over: never hold more than a small window of the data at once.

This lesson turns that idea into concrete habits: stream instead of buffer, respect backpressure, batch your writes, push CPU-heavy work to worker threads, and checkpoint long jobs so they can resume after a crash. It builds on Streams Fundamentals, Pipelines and Backpressure, Worker Threads and SQLite with node:sqlite.

Buffering versus streaming

Buffering means reading everything into memory, then working on it. await readFile(path, 'utf8') followed by .split('\n') holds the file as a string, then holds all the lines as an array: easily two or three times the file size. Streaming reads a chunk, handles it, throws it away, and reads the next one. Memory use stays flat whether the file is 1 MB or 100 GB.

Node's default old-space limit is a few gigabytes (our editor runs with 256 MB), and a single string cannot exceed roughly 512 MB (V8's maximum string length). So buffering has a hard ceiling long before the disk is full.

The next example creates a file of 50,000 lines with a stream (so the generator itself never builds the whole thing), then counts lines and sums a column two ways.

Node.js Buffered versus streamed processing of the same file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Both give the same answer. The difference only shows up when the file grows: the buffered version needs memory proportional to the file, the streamed version needs a few kilobytes. Prefer streaming whenever the input size is not under your control, which is almost always the case for uploads, exports and logs.

A good rule of thumb: if the data comes from a user, a partner, or a database table that grows, treat it as unbounded and stream it. Buffer only when you know the size is small and fixed, such as a config file.

Backpressure: the producer must wait

A fast producer and a slow consumer is the classic recipe for a memory leak. If you keep calling writable.write() without looking at its return value, Node queues every chunk in memory until the slow side catches up. write() returns false when the internal queue passes highWaterMark; that is the signal to stop and wait for the 'drain' event.

You rarely do this by hand because pipeline() and for await handle it. But you should have seen it once, to know what they do for you. Here a deliberately slow writable accepts one chunk per tick, and we count how often the producer has to pause.

Node.js Counting backpressure pauses by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The producer wrote 8,000 bytes in total, but the queue never held more than about 1,200 at a time. Remove the await once(sink, 'drain') line and the queue would grow to the full 8,000 bytes. Scale that to 8 GB and you can see why ignoring backpressure kills servers.

Two classic mistakes: calling stream.write() in a loop and ignoring its result, and using readable.on('data', ...) with an async handler that awaits slow work. Events do not wait for your promises. Use for await (const chunk of readable) or pipeline() so each chunk is finished before the next one is requested.

Batch your writes

Streaming protects memory on the way in. On the way out, the other trap is making one tiny operation per record: one INSERT per row, one HTTP request per item. Each operation has a fixed cost (a transaction commit, a network round trip) that dwarfs the useful work. Batching groups many records into one operation.

With SQLite the effect is dramatic because every standalone INSERT is its own transaction with its own disk sync. Wrapping 1,000 inserts in one transaction pays that cost once. The example streams 5,000 generated records into an in-memory database in batches of 1,000. (node:sqlite is built in; in Node 24 it is still flagged experimental and prints a warning on stderr.)

Node.js Streaming records into SQLite in batches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Pick a batch size by measuring on your real system, but a few hundred to a few thousand rows is a common sweet spot. Larger batches use more memory and make a failure lose more work; smaller ones waste round trips. For HTTP APIs, check whether the service has a bulk endpoint, and mind its payload limits.

Move CPU-heavy work to worker threads

Streaming keeps memory low, but it does not help if each chunk needs heavy computation: parsing, compressing, image resizing, scoring. That work runs on the main thread and blocks the event loop, so every other request waits. The fix is a worker pool: a fixed number of worker threads, each handling one job at a time, with a queue in front.

The example below counts primes in six ranges using two workers. One file plays both roles: isMainThread tells them apart. Results are collected by index, so the output order is deterministic even though workers finish in any order.

Node.js A small worker pool with ordered results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three design points matter. First, the pool size is fixed (usually the number of CPU cores), so a flood of jobs cannot spawn a flood of threads. Second, jobs are small and independent, which keeps workers busy and results easy to merge. Third, each message is copied between threads, so send references (file offsets, ids) rather than megabytes of data when you can. Our sandbox has one CPU, so you will not see a speedup here; on a real machine the pool scales with cores.

Make long jobs resumable

A job that processes 40 million rows will, sooner or later, be interrupted by a deploy, a crash, or a network blip. If it has to start over each time it may never finish. The fix is a checkpoint: periodically save how far you got, and on start-up read it and continue from there. Two properties make this safe: the checkpoint is written after the work it covers, and the work is idempotent (repeating a record is harmless), because a crash can happen between the work and the checkpoint.

This example processes numbered lines, "crashes" after line 7, and then resumes. The checkpoint holds the last fully processed line number.

Node.js Checkpointing a job so it can resume

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

In a real system, writing the checkpoint after every record is too slow; write it every N records or every few seconds, and accept that up to N records may be repeated after a crash. That is exactly why idempotency matters. To make the write itself crash-safe, write to a temporary file and rename it over the old one.

Chunked and resumable uploads

The same ideas apply to uploads. Do not ask a client to send 4 GB in one request. Let it send fixed-size chunks (say 5 MB), each with an offset or index. The server appends each chunk to a temporary file, answers with the number of bytes it has so far, and a client that lost its connection asks "how far did you get?" and continues from there. On the server side you pipe each request body straight to disk (pipeline(req, createWriteStream(path, { flags: 'a' }))) so memory stays flat, and you verify a checksum at the end. See Request Bodies, JSON and Forms and Hashing and HMAC.

Pitfalls professionals watch for

  • JSON.parse on a huge file. JSON is not streamable with the built-in parser. For big data, use JSON Lines (one JSON object per line) and parse line by line, or a streaming parser library.
  • Accumulating results. Streaming the input does not help if you push every output record into one array. Write results out as you go, or aggregate (counts, sums) instead of collecting.
  • Unbounded concurrency. Promise.all(hugeArray.map(fetchOne)) starts everything at once. Limit concurrency as in Concurrency Control.
  • Reading with a tiny highWaterMark. The default of 64 KB for file streams is a good start; raising it to 256 KB or 1 MB can help sequential throughput, but measure first.
  • Forgetting errors. An error on any stream in a chain must stop the whole chain. pipeline() does that; manual .pipe() does not.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does streaming a 10 GB file keep memory use flat?

    A stream reads a chunk, lets you process it and moves on, so memory depends on the chunk size, not on the file size.
  2. 2. What does writable.write(chunk) returning false mean?

    The chunk is still accepted and queued. The false value is advice to stop writing until 'drain', otherwise the queue keeps growing in memory.
  3. 3. Why batch database inserts into transactions?

    Each standalone insert is its own transaction. Grouping rows amortizes the commit cost, at the price of losing more work if a batch fails.
  4. 4. A resumable job crashes between processing a record and saving its checkpoint. What property keeps the data correct?

    Because the checkpoint is saved after the work, a crash can cause the last records to be processed again on restart. Idempotent work makes that safe.
  5. 5. Which is the best way to handle JSON data too big for memory?

    JSON.parse needs the whole document as a string. One JSON object per line can be read and parsed incrementally with a line reader.

Summary

  • Treat any input whose size you do not control as unbounded and stream it; buffering has hard limits.
  • Respect backpressure: pipeline() and for await do it for you, manual write() loops must wait for 'drain'.
  • Batch writes to databases and APIs to amortize per-operation cost, and flush the final partial batch.
  • Offload CPU-heavy chunks to a fixed-size pool of worker threads; send references, not bulk data.
  • Checkpoint long jobs after the work they cover and make the work idempotent so a crash only costs a little repeated effort.
  • Use chunked uploads, JSON Lines, and bounded concurrency to keep every stage of the pipeline small.

Next: Scalability and Reliability Patterns