Skip to content
Tutorials Center
My learning

Topics

Lesson 118 of 133 ยท Projects

Project: Streaming Log Processor

Build a log analyzer in Node.js that reads huge files with streams in constant memory. Parse, aggregate, report, handle bad lines and errors, add gzip, tested with a seeded generator.

Every server writes logs, and sooner or later someone asks a question that only the logs can answer. How many requests failed yesterday? Which page is the slowest? Which client is hammering the API? The log file is usually large: hundreds of megabytes, sometimes many gigabytes. Reading it all into a string would exhaust memory long before the answer appears.

Node.js streams are made for this. In this project you build a log processor that reads a file piece by piece, parses each line, updates a few counters, and prints a report. It never holds more than a small chunk of the file in memory, so it handles a 20 MB log and a 20 GB log with the same memory use. You will build it in stages, from the parser to a small command-line tool with gzip support, and everything runs here in the editor with a seeded log generator, so the numbers are the same on every run.

The log format

Real access logs come in many formats. We use one simple, space-separated line per request, which is easy to parse and similar in spirit to common web server formats:

TEXT
2026-01-01T00:00:03.000Z 203.0.113.9 GET /api/users 200 1234 17ms

The fields are the time (ISO 8601, UTC), the client IP, the HTTP method, the path, the status code, the response size in bytes, and the time taken in milliseconds. The addresses in 203.0.113.0/24 are reserved for documentation, so nothing in the lesson refers to a real machine.

Real logs also contain damaged lines: a crash cuts a line in half, two processes write at once, someone adds a comment. A good processor must count and skip such lines instead of crashing on them, and the generator below puts a damaged line at every 250th position to test exactly that.

Stage 1: a parser and a repeatable log

Start with two pure pieces that need no streams: a function that parses one line into an object (or returns null), and a generator that creates test logs. The generator uses a small seeded random number generator (seededRandom, a well-known 32-bit mixer), so generateLog(100) is the same text on every machine. Repeatable test data is half of testing a data tool.

Node.js Stage 1: generate and parse log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first block prints five generated lines. Look at the times: they never go backwards, and two requests can share a second, as in a real busy log. Then parseLine turns the sample into an object, and returns null for a damaged line and for a line that looks right but has an impossible date (2026-13-45). Checking Date.parse matters, because a regular expression alone cannot know that month 13 does not exist.

Two small decisions are worth copying. The parser returns null instead of throwing, because bad lines are normal data here and not exceptional events, and a throw per line would be slow when 1% of a billion lines are bad. And the regular expression is anchored with ^ and $ and uses \S+ and explicit digit classes, so it cannot backtrack catastrophically on a malicious line (see web security for why that matters).

Stage 2: turn chunks into lines

A file stream delivers chunks: by default up to 64 KiB each, cut wherever the 64 KiB boundary happens to fall, usually in the middle of a line. If you process each chunk as if it were a list of whole lines, you will parse the end of one line glued to nothing, and the start of the next line glued to nothing, and silently lose both. The cure is the same carry-over trick used in stream patterns: keep the unfinished tail of a chunk and put it in front of the next one.

That is what splitLines does. It is an async generator: it receives the source stream as an async iterable and yields complete lines. The next example tests it with chunks cut in awkward places, and shows the second classic pitfall: a multi-byte character such as โ‚ฌ (three bytes in UTF-8) cut in half by a chunk boundary.

Node.js Lines across chunk boundaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first call shows that a line can span three chunks (cd + ef, then g + h), that a final line without a newline is still delivered, and that an empty input yields nothing. The second half shows why the stage 3 code asks the file stream for { encoding: 'utf8' }: that option makes the stream decode bytes with a StringDecoder, which holds back an incomplete character until its remaining bytes arrive. Without it, chunk.toString() turns the cut character into three useless replacement characters (๏ฟฝ) instead of one โ‚ฌ. See text encoding.

Stage 3: aggregate in constant memory

Now the heart of the tool. The Stats class keeps a few counters, never the entries themselves, and that decision is what makes the memory use flat. Each entry updates:

  • the request, byte and malformed-line totals;
  • a count per status class (2xx, 4xx, ...) and per path;
  • a count per minute, from which the busiest minute is picked;
  • a Set of client IPs;
  • a latency histogram.

The histogram deserves a word. To report the 95th percentile exactly you must keep every latency value and sort them, which is memory proportional to the file size, the very thing we are avoiding. A histogram with fixed buckets (<= 25 ms, <= 50 ms, ...) uses a handful of numbers whatever the input size, and answers "which bucket holds the 95th percentile?". The price is precision: the report says <= 750 ms, not 712 ms. For monitoring that is usually the right trade; real tools use the same idea with finer buckets (HDR histograms, t-digests).

Not every counter is bounded. The paths map grows with the number of different paths, the IP set with the number of different clients, and the minute map with the length of the period. In this log those stay small (five paths, 40 IPs, about eight hours), but a log with millions of unique URLs, such as one that includes query strings with random tokens, would grow the map without limit. Normalize paths before counting, cap the map size, or use an approximate counter (HyperLogLog for distinct counts, count-min sketch for heavy hitters).

The pipeline itself is pipeline(source, splitLines, async function (lines) { ... }). The first step is the stream, the middle step is the generator, and the last step is an async function that consumes the lines and updates the stats. stream/promises pipeline wires them together, applies backpressure (the file is read only as fast as the consumer processes it) and destroys everything if any step fails. Write it by hand with .pipe() and a failure in the middle would leave the file descriptor open.

The program below writes a 20,000-line log to access.log with a stream (the generator is the source, so even creating the file never holds it in memory), then analyzes it. It also records the largest chunk the file stream produced, as evidence for the memory claim.

Node.js Stage 3: stream a 20,000-line log into a report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Reading the report: of the 20,000 lines, 19,920 are requests and 80 are damaged (every 250th line, so exactly 80). The status classes add up to 19,920. Nearly 17,000 requests succeeded and 615 were server errors. / is the most visited path with 6,121 hits. The median latency falls in the <= 250 ms bucket and the 95th percentile in <= 750 ms.

The line that proves the design is largest chunk held in memory: 65536 characters. The file is more than a megabyte, yet the processor never held more than 64 Ki characters of it at a time, and the file could be a thousand times larger without changing that number. That is constant memory, and it is the reason to use streams for files.

Streams have a cost per chunk, not per byte, so the chunk size matters for speed. The default 64 KiB is a good start. For huge files, createReadStream(path, { highWaterMark: 1024 * 1024 }) reads 1 MiB at a time and cuts the number of system calls by 16. Measure on your own data before tuning (see performance tuning).

Stage 4: failing well

A tool that works on a perfect file is half a tool. Two kinds of failure matter here, and they need opposite treatment.

Bad data is expected. Damaged lines are counted in stats.bad and reported as malformed, but the run continues. If more than, say, 5% of lines are bad, the right behavior is a warning on stderr and a non-zero exit code, since the file is probably not a log in the format you assumed.

A broken source is fatal. If the file does not exist or cannot be read, there is nothing to report. Because pipeline from stream/promises returns a promise, the error becomes an ordinary rejection that you can catch. If you do not, you get this:

Node.js A missing log file (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The process dies with exit code 1 and prints the error with its code (ENOENT, "no such file or directory") and the path. Node 24 shows no stack frames here, because the error came from the operating system through the stream and not from a line of your code. A user of your tool does not want a crash dump. They want a short message and a meaningful exit code, so catch it where you started the pipeline:

Node.js The same call, handled

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Setting process.exitCode = 2 instead of calling process.exit(2) lets pending output flush and the process end by itself. Use distinct codes for distinct problems (for example 1 for "bad input data", 2 for "cannot read the file") so scripts that call your tool can react (see process).

Other failures are possible later in a pipeline, such as a read error halfway through a file on a flaky disk, or a handler that throws on an unexpected entry. They all surface through the same rejection, and pipeline has already closed the file for you by then. This is the main reason to prefer pipeline over chaining .pipe() calls.

Stage 5: a small command-line tool

Now wrap the pieces in the shape of a real tool. The parseArgs function from node:util (see CLI tools) reads options such as --top 5, --format json and --gzip. We give it an explicit args array because the editor cannot pass command-line arguments; in a real script you would omit args and it would read process.argv.

The new idea is composition. The --gzip option first compresses the log into access.log.gz, then analyzes it by piping the file through createGunzip(). The analyzer neither knows nor cares: it only receives a stream of text. Reading a .gz log is just one more step in front of the same pipeline, and the memory stays constant, because decompression is also a stream (see zlib).

Node.js Stage 5: a command-line tool with gzip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The tool analyzed a different, smaller log (5,000 lines, seed 99): 4,980 requests, 20 damaged lines, and the two busiest paths printed with console.table. Switch --format to json in the args array to get the machine-readable report from before. Offering both formats is a good habit: people read tables, other programs read JSON.

Ideas for taking it further

  • Filters. Add --since and --status 5xx options and skip non-matching entries before they reach Stats. A filter is just another generator in the pipeline.
  • Follow mode. tail -f style monitoring needs a file watcher and reading from the last offset (see file streams and the start option of createReadStream).
  • Parallelism. One thread parses a few hundred thousand lines per second. For multi-gigabyte logs, split the file into byte ranges that begin at line boundaries and give each range to a worker thread, then merge the Stats objects (counters are easy to add together; histograms add bucket by bucket).
  • Output sinks. Write the report to a file with fs.writeFile, or stream a CSV with one row per minute (see CSV and text processing).
  • Tests. The parser, the splitter and Stats are pure, so they are ideal for the node:test runner. The seeded generator gives you a fixed, large input whose report you can pin in a test.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can a log analyzer built on streams process a file larger than the available memory?

    A read stream delivers the file chunk by chunk, and the analyzer updates fixed-size counters. Memory use depends on the chunk size and the counters, not on the file size.
  2. 2. What goes wrong if you split each chunk on newlines without carrying over the unfinished tail?

    A chunk boundary can fall anywhere, usually in the middle of a line. Carrying the unfinished part over and prepending it to the next chunk rebuilds the full line.
  3. 3. Why does the stats code use a latency histogram instead of a list of all latencies?

    Exact percentiles need every value. Fixed buckets use constant memory and answer "which bucket holds the 95th percentile", which is precise enough for monitoring.
  4. 4. What is the advantage of pipeline from node:stream/promises over chaining .pipe() calls?

    pipeline handles backpressure and cleans up every stream when one fails, and the promise version lets you use try/catch. With .pipe() an error in the middle leaves the other streams open.
  5. 5. How does the tool read a gzip-compressed log with the same analyzer?

    Streams compose. createGunzip() turns compressed bytes into text chunks, and the rest of the pipeline neither knows nor cares where the text came from.

Summary

  • Streams let a tool process files of any size with constant memory: a chunk at a time plus a few counters.
  • Rebuild lines across chunk boundaries by carrying the unfinished tail, and decode with an encoding so multi-byte characters are not cut.
  • Treat bad lines as data: parse to null, count them, keep going. Treat a missing or unreadable source as fatal, with a clear message and an exit code.
  • Prefer fixed-size summaries (counters, histograms) over lists. Watch maps and sets whose size depends on the input.
  • stream/promises pipeline joins the steps, applies backpressure, cleans up on failure and gives you one promise to await.
  • Streams compose: gzip, filters and new sources are extra steps, not rewrites.
  • A seeded generator gives repeatable test data and repeatable output.

Next: Project: Background Job Queue