Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 39 ยท TOML, CSV and Other Formats

JSON Lines and Binary Formats

Learn JSON Lines for streaming records and the binary alternatives MessagePack, CBOR, BSON and Protocol Buffers - how they encode data, what they save, and how to build a tiny encoder in Node.

JSON and CSV are text formats: you can open them in any editor. That is a great property for configuration and debugging, but text has costs. A JSON array must be read to its closing bracket before you know it is valid, which is awkward for logs and exports with millions of records. And every number is spelled out in digits, every key is repeated in every object, which wastes space and CPU on big or frequent messages. This lesson covers two answers: JSON Lines keeps the text but makes it streamable, and binary formats trade readability for size and speed.

JSON Lines (NDJSON)

JSON Lines (also called NDJSON, newline-delimited JSON) stores one complete JSON value per line, usually an object, separated by \n. There is no surrounding array and no commas between records.

TEXT
{"id":1,"event":"login","user":"ada"}
{"id":2,"event":"view","page":"/docs"}
{"id":3,"event":"logout","user":"ada"}

The benefits come from the line structure:

  • Append-only. Writing a new record is one write of a line, with no need to rewrite a closing bracket. Logs and event streams use this.
  • Streaming. A reader processes one line at a time and keeps only one record in memory.
  • Tool-friendly. head, tail, grep, wc -l and split all work on records. A damaged line affects only itself.
  • Parallel. A big file can be cut at line boundaries and processed by many workers.

The rules: each line is valid JSON, no raw line breaks inside a record (a JSON string escapes them as \n, so compact JSON.stringify output is always safe), UTF-8 encoding, and a trailing newline is allowed. The usual file extensions are .jsonl and .ndjson. Many databases and services import and export it: BigQuery, Elasticsearch bulk API, OpenAI fine-tuning files, and countless log shippers.

Node.js Write and stream-read JSON Lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice that the record with a newline in its note is still a single line: JSON.stringify wrote \n as two characters.

Why binary formats?

A binary format encodes data as bytes instead of characters. The advantages are real:

  • Smaller. A 32-bit integer takes at most 4 bytes, not up to 10 digits plus a comma; keys can be replaced by numbers.
  • Faster. No text scanning, no number parsing from digits.
  • Richer types. Raw bytes (JSON would need base64), timestamps, 64-bit integers, binary blobs.

The disadvantages: you cannot read the data without a tool, debugging is harder, and some formats need a separate schema file.

MessagePack and CBOR

MessagePack and CBOR (RFC 8949) are binary "JSON-like" formats. They have the same data model (maps, arrays, strings, numbers, booleans, null) and need no schema; every value carries a small type-and-length header. CBOR adds tags (dates, bignums) and is the basis of IoT standards such as COSE and WebAuthn. MessagePack is used in Redis scripting and many RPC systems. BSON is MongoDB's binary JSON with extra types such as ObjectId and Date.

For example, MessagePack encodes the small integer 5 as the single byte 0x05, the string "hi" as 0xa2 0x68 0x69 (type+length byte, then UTF-8), and a map with two entries as 0x82 followed by the keys and values.

Node.js A tiny MessagePack-style encoder (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The binary version is a bit more than half the size here, and the saving grows with numeric data. Note how keys are still strings: MessagePack and CBOR repeat keys just like JSON. Schema-based formats go further.

Protocol Buffers and schema-based formats

Protocol Buffers (protobuf), Avro and Thrift describe messages in a schema file (.proto). Fields are identified by numbers, not names, so the wire data never repeats key names. A protobuf message for {id: 300} with id as field 1 is just three bytes: a tag byte (field 1, varint type), then the varint for 300. The schema is required to read it, and a compiler generates classes for each language. This is the basis of gRPC. Because fields are numbered, you can add fields later without breaking old readers, as long as you never reuse a number.

A varint stores an integer in 7-bit groups, least significant first, with the high bit of each byte meaning "more bytes follow". Small numbers take one byte.

Node.js Protobuf-style varints and a field by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The 300 is encoded as ac 02, matching the protobuf documentation. These are exactly the primitives real libraries are built on.

Choosing

flowchart TD
  A["Need to store or send records"] --> B{"Humans read it often?"}
  B -- "yes" --> C["JSON / YAML / CSV"]
  B -- "no" --> D{"Schema and codegen acceptable?"}
  D -- "yes" --> E["Protobuf / Avro (fast, compact, evolvable)"]
  D -- "no" --> F["MessagePack / CBOR (JSON-like, binary)"]
  C --> G{"Millions of records or streams?"}
  G -- "yes" --> H["JSON Lines"]

A first decision tree; see Choosing a Format for more.

Start with JSON; move to a binary format only when measurement shows size or speed is a real problem. Compression (gzip, brotli, zstd) over plain JSON often closes much of the size gap with far less change.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What defines JSON Lines?

    Each line is a separate JSON document, so files can be appended to and read one record at a time.
  2. 2. Why can a JSON string with a line break still be one JSON Lines record?

    Compact JSON.stringify output never contains a raw newline; control characters inside strings are escaped.
  3. 3. Which format needs a separate schema to decode the bytes?

    Protobuf messages use field numbers and are decoded using the .proto definition. MessagePack and CBOR are self-describing.
  4. 4. What is a varint?

    Varints use 7 bits per byte with a continuation bit, so 1 takes one byte and 300 takes two.

Summary

  • JSON Lines puts one JSON value per line, enabling appends, streaming and line-based tools.
  • Binary formats are smaller and faster but unreadable without tools.
  • MessagePack and CBOR are self-describing JSON-like binaries; protobuf and Avro use schemas and numbered fields.
  • Varints encode small integers in few bytes.
  • Measure before switching: compressed JSON is often good enough.

Next: Markdown Syntax: The Complete Guide