Skip to content
Tutorials Center
My learning

Topics

Lesson 91 of 132 ยท Standard Library Tour

JSON, Serialization and Text Formats by Hand

Build a JSON value type, printer and parser in plain Rust, write a CSV reader with quoting, learn escaping rules, and see how serde derives Serialize and Deserialize.

Programs constantly need to turn in-memory data into text and back: configuration files, API payloads, logs, spreadsheets. This is called serialization (data to text or bytes) and deserialization (the reverse). In everyday Rust you almost never write this by hand; you use the serde crate. But writing a small JSON printer and parser yourself is one of the best exercises in the language. It uses enums, recursion, Display, iterators, Result and error types all at once, and afterwards serde stops being magic.

Our editor only has the standard library, so this lesson builds everything from scratch: a Value enum for JSON, a printer with correct string escaping, a recursive-descent parser, a pretty printer, and a CSV reader that handles quotes. At the end you will see what the same job looks like with serde and serde_json, in read-only snippets, so you know what to reach for in real projects.

Modelling JSON as an enum

JSON has exactly six kinds of value: null, booleans, numbers, strings, arrays and objects. That is a perfect fit for a Rust enum, where each variant carries its own data. Arrays and objects contain more values, so the type is recursive; Vec already stores its elements on the heap, so no Box is needed.

For objects we use BTreeMap<String, Value>. JSON objects have no meaningful order, and a sorted map gives deterministic output, which is also what you want in tests. (A HashMap would print its keys in a different order each run.)

Rust A Value type with a Display printer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Notes on this printer:

  • prints an f64 that holds a whole number as 36 (no .0), which is exactly what JSON wants. f64 cannot represent NaN or infinity in JSON though; a production printer must reject or map them.
  • Escaping is not optional. A quote or backslash inside a string must be escaped, and control characters below U+0020 must be written as \n, \t or \u00XX. Skip this and you produce invalid JSON, or worse, a way for attackers to inject extra fields.
  • Writing into a Formatter with write_str avoids building a temporary String for every node.

Writing the parser: a tour of recursive descent

A recursive-descent parser has one function per grammar rule, and those functions call each other the same way the rules nest. JSON's grammar fits on a napkin:

TEXT
value  = null | true | false | number | string | array | object
array  = "[" [ value { "," value } ] "]"
object = "{" [ pair { "," pair } ] "}"
pair   = string ":" value

So we need parse_value, which looks at the next character and decides which rule to follow, plus parse_array, parse_object, parse_string and parse_number.

The tool for looking ahead is Peekable<Chars>: peek() shows the next character without consuming it. We also record the position so errors can say where the problem is.

Errors deserve their own type. A ParseError with a message and a character offset is much friendlier than unwrap on a None.

Rust A complete JSON parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

How it works, step by step:

  1. parse_value skips whitespace and peeks at one character. That single character is enough to pick the rule: n, t, f start literals, " a string, [ an array, { an object, and a digit or - a number. This is called an LL(1) grammar: one character of lookahead decides everything.
  2. parse_array consumes [, then loops: parse a value, then either a comma (continue) or ] (done). It handles the empty array before the loop, which is why [1,] correctly fails with "unexpected character ']'".
  3. Objects work the same way with key : value pairs.
  4. The top-level parse function insists that nothing but whitespace follows the value, so 1 2 is rejected.

Every parse function returns Result<_, ParseError> and uses ? to forward failures, so the first error unwinds the whole recursion in one step. The result is a parser with no unwrap and no panics, even for garbage input.

Two honest limitations of this toy: it accepts a few things strict JSON forbids (leading zeros such as 007 and a trailing dot as in 1. both reach f64::parse and succeed), and deeply nested input such as 100,000 open brackets will overflow the stack through recursion. Real parsers enforce the exact number grammar and a depth limit. When you parse untrusted data, always add a depth counter.

Pretty printing with indentation

The compact printer is right for machines. People want indentation. A pretty printer is a recursive function that carries the current depth:

Rust A pretty printer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Passing &mut String as an output buffer is the idiomatic way to build large text recursively: one allocation grows as needed, instead of every level creating and concatenating its own String.

Using Rust's {:?} for strings is a shortcut that is close to JSON but not identical (for example it writes \u{1b} for control characters where JSON needs \u001b). The first lesson's write_escaped is the correct approach.

CSV by hand

CSV looks trivial and is not. The rules (as most tools implement them) are:

  • Fields are separated by commas, records by newlines.
  • A field containing a comma, a quote or a newline must be wrapped in double quotes.
  • Inside a quoted field, a quote is written twice: "She said ""hi""".

Splitting on commas works until the first quoted comma. A tiny state machine over the characters is the right tool. This reader handles quotes and escaped quotes, and the writer is the inverse:

Rust A CSV reader and writer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Notice std::mem::take(&mut current): it moves the finished field out and leaves an empty String behind, so we reuse the variable without cloning. The reader works line by line, which means it does not support newlines inside quoted fields. To support them, the state machine must run over the whole input rather than line by line. For serious CSV work use the csv crate, which handles that, plus encodings, headers and streaming.

Text formats: choosing and escaping

Whatever format you write, the core rule is the same: escape on output, unescape on input, and never build structured text with plain string concatenation of untrusted data. Each format has its own escape rules:

Format Must escape Notes
JSON ", \, control characters Numbers must be finite. Strings are Unicode.
CSV " doubled; quote fields with ,, ", newline Dialects differ (semicolon separators, etc.).
HTML/XML <, >, &, quotes in attributes Wrong escaping causes cross-site scripting.
Shell/SQL Avoid it: pass arguments or parameters separately String-built commands lead to injection.

For a simple key=value configuration format, you can parse with split_once('='), which returns Option<(&str, &str)> and keeps the code short:

Rust
for line in "host = example.org\nport = 8080".lines() {
    if let Some((key, value)) = line.split_once('=') {
        println!("{} -> {}", key.trim(), value.trim());
    }
}

serde: how real projects do it

The serde crate separates two concerns: data types that know how to describe themselves (Serialize, Deserialize) and formats that know how to read and write (serde_json, toml, csv, bincode, and many more). You derive the traits once and get every format for free. These snippets need crates.io, so they are read-only here.

TOML
# Cargo.toml (read-only: needs crates.io, not available in our editor)
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
Rust
// Read-only: requires the serde and serde_json crates.
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct User {
    name: String,
    age: u32,
    #[serde(default)]
    admin: bool,
    #[serde(rename = "emailAddress", skip_serializing_if = "Option::is_none"Option::is_none")]
    email: Option<String>,
}

fn main() -> Result<(), serde_json::Error> {
    let user: User = serde_json::from_str(r#"{"name": "Ada", "age": 36}"#)?;
    println!("{user:?}");
    println!("{}", serde_json::to_string_pretty(&user)?);
    Ok(())
}

What the derive gives you, compared with our hand-written version:

  • Typed results. You get a User, not a loose Value tree you must pick apart. A missing age or a string where a number belongs is a clear error such as "missing field age".
  • Attributes to rename fields, supply defaults, skip fields or flatten structures.
  • No intermediate tree. serde deserializes straight into your struct, which is faster and uses less memory than building a Value first.
  • Many formats from the same derive.

Our Value enum is still useful: serde_json::Value is the same idea and is the right choice when the shape of the data is not known in advance.

Common mistakes

  • Splitting CSV with line.split(',') and corrupting quoted fields.
  • Forgetting to escape strings on output and producing invalid JSON.
  • Parsing numbers with parse::<f64>() and trusting the result for money; use integers (cents) for currency.
  • Recursing without a depth limit on untrusted input.
  • Reporting errors without a position, which makes bad input impossible to debug.
  • Using unwrap inside a parser; a parser's whole job is to expect bad input.

See also: Display and From for implementing Display, custom error types, and the next lessons on randomness and hashing and hand-written parsers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the example use BTreeMap instead of HashMap for JSON objects?

    JSON object order is not significant, but a HashMap iterates in a random order that changes between runs. A BTreeMap keeps keys sorted, so output is stable.
  2. 2. What is the purpose of Peekable in the parser?

    peek() lets parse_value decide between null, string, array and so on by inspecting one character of lookahead without removing it.
  3. 3. Which CSV field must be wrapped in quotes when written?

    A field containing a comma, a double quote or a newline must be quoted; any quote inside it is doubled.
  4. 4. How is a double quote written inside a quoted CSV field?

    CSV escapes a quote by doubling it, so She said "hi" is written "She said ""hi""".
  5. 5. What advantage does serde's derive have over building a loose Value tree?

    With #[derive(Deserialize)] the data lands in a struct with the right field types, and format-independent attributes handle renames and defaults.

Summary

  • Serialization turns data into text or bytes; deserialization reverses it. In real projects use serde; writing a parser by hand teaches how it works.
  • JSON maps naturally to a recursive enum Value with BTreeMap objects.
  • Always escape strings on output; control characters become \n, \t or \u00XX.
  • A recursive-descent parser has one function per rule, uses Peekable<Chars> for lookahead, and returns Result with a position-carrying error type.
  • Limit recursion depth when parsing untrusted input.
  • CSV needs a quote-aware state machine; splitting on commas is wrong.
  • serde with serde_json (read-only here) deserializes directly into typed structs.

Next: Randomness, Hashing and Math