Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 39 ยท Projects and Practice

Project: A JSON Lines Log Analyser

Build a streaming log analyser in Node - parse JSON Lines, skip bad lines, validate records, aggregate counts and percentiles, and publish the report as CSV, JSON and a Markdown table.

Logs are the most common data you will ever analyse, and the best modern format for them is JSON Lines: one self-describing record per line. In this project you build a small analyser that reads a stream of request logs, copes with damaged lines, validates each record, computes useful statistics, and writes the result in three formats: CSV for spreadsheets, JSON for programs, and a Markdown table for a report or a pull-request comment. It exercises parsing, validation, streaming, escaping and conversion in a single, realistic tool.

The data

Each line is one HTTP request:

TEXT
{"ts":"2025-03-14T10:00:01Z","method":"GET","path":"/products","status":200,"ms":42}

Real logs are messy: truncated lines, a missing field, a string where a number belongs, blank lines. The analyser must count these problems, not crash on them.

flowchart LR
  A["JSON Lines stream"] --> B["parse line"]
  B -->|"invalid JSON"| X["bad-line counter"]
  B --> C["validate record"]
  C -->|"wrong shape"| X
  C --> D["aggregate: per path, per status class"]
  D --> E["percentiles"]
  E --> F["CSV"]
  E --> G["JSON"]
  E --> H["Markdown table"]

Invalid input is counted and skipped; valid records feed the aggregation.

Step 1: read lines as a stream

readline gives one line at a time, so memory use stays constant however large the file. In the sandbox the "file" is generated in memory with a deterministic pseudo-random sequence (no Math.random, so the output is always the same), and a few corrupt lines are mixed in on purpose.

Step 2: the analyser

Node.js Log analyser with CSV, JSON and Markdown output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the first line of the output: the analyser found 200 valid records and reported exactly what it skipped. Without such a count, a log analyser that silently ignores bad lines would give confident but wrong answers.

Design notes

  • Parse, then validate. JSON.parse only proves the line is JSON. validate proves it has the shape the program needs; the two failures are counted separately because they have different causes (truncation versus a producer bug).
  • Percentiles, not averages. Mean latency hides slow requests; p95 says "95% of requests were at least this fast". The nearest-rank method used here is simple and deterministic.
  • Memory. The code keeps all latencies per path to compute percentiles exactly. For truly huge inputs use an approximate structure (t-digest, HDR histogram) or fixed-size buckets.
  • One data set, three outputs. The rows array is the single source; each writer only formats it. Adding XML or YAML output means adding one more function.
  • Escaping per format. CSV cells quote and guard against formulas; Markdown cells escape pipes; JSON is serialised by JSON.stringify. Never share one escaping function across formats.

Extensions

  1. Group by hour using the ts field, and add a time-series CSV.
  2. Read the file from process.argv[2] with fs.createReadStream and write the three reports to files.
  3. Accept a --since filter and use a JSON Schema (Command-Line Data Tools) for validation.
  4. Emit the Markdown as a GitHub-flavoured step summary in CI (GitHub Flavored Markdown).
  5. Flag any path whose p95 is more than twice the overall p95.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the analyser count invalid lines instead of crashing?

    Robust tools skip and report bad records so results stay useful and the data quality is visible.
  2. 2. Why is p95 more informative than the mean for latency?

    A few slow requests barely move the mean but strongly affect user experience; percentiles expose them.
  3. 3. Why use readline over a stream for large logs?

    Streaming never holds the whole file in memory.
  4. 4. Why does each output format have its own escaping?

    CSV needs quotes and formula guards, Markdown needs escaped pipes, and JSON has its own string rules.

Summary

  • JSON Lines logs are parsed line by line; damaged lines are counted, not fatal.
  • Validate shape after parsing; report what was skipped.
  • Aggregate with maps; summarise latency with percentiles.
  • Write the same rows as CSV, JSON and Markdown using format-specific escaping.
  • Streaming keeps memory flat; extend with time buckets, files and schemas.

Next: Project: A Static Page Builder from Markdown