Skip to content
Tutorials Center
My learning

Topics

Lesson 95 of 132 ยท Standard Library Tour

Serialization and Data Formats by Hand

Build a serde-style Serialize trait and Serializer formats in plain Rust, write JSON and a binary format from the same code, and see how derive and TOML fit in.

Almost every Rust program that talks to the outside world turns values into bytes or text and back again: a config file, a JSON API, a cache entry, a network message. In real projects you use the serde crate for this, and a single #[derive(Serialize, Deserialize)] line gives your struct support for dozens of formats. It feels like magic, and magic is hard to debug. This lesson takes the magic apart by building a miniature version using only the standard library.

The earlier lesson JSON, Serialization and Text Formats by Hand wrote a JSON value type and a parser. This one takes the other half of the story: the design that lets one type be written in many formats without the type knowing any of them. You will build two traits, a JSON format and a binary format on top of them, and a macro_rules! stand-in for derive. At the end you will see the real serde code, which will now look very familiar.

The idea: a data model between types and formats

Suppose you have M data types and N formats (JSON, TOML, YAML, binary, ...). Writing each pair by hand needs M x N pieces of code. Serde avoids that with an intermediate vocabulary called the data model: a small set of concepts such as "boolean", "integer", "string", "optional value", "list" and "map with named fields". Then:

  • Every data type implements Serialize once, saying how it is described in the data model.
  • Every format implements Serializer once, saying how each data model concept is written.

Now any type works with any format, with only M + N pieces of code.

flowchart LR
    A["Config struct"] --> S["Serialize trait<br/>(describes itself)"]
    B["Vec of numbers"] --> S
    C["Option of text"] --> S
    S --> M["Serializer trait<br/>(the data model)"]
    M --> J["JSON text"]
    M --> T["TOML text"]
    M --> X["Binary bytes"]

Types talk to the Serializer trait; formats implement it. Neither side knows the other.

In our version the data model is the Serializer trait below: eleven methods are enough for booleans, integers, floats, strings, optional values, lists and structs. The Serialize trait has one method that receives a serializer and calls the matching methods on it. Real serde uses a generic parameter (fn serialize<S: Serializer>(&self, s: S)) so everything is monomorphized and inlined; we use &mut dyn Serializer to keep the signatures short. The structure is the same.

Writing JSON from the data model

The JSON serializer needs state because JSON has separators: a comma between list items and between struct fields, but not before the first one. It keeps a stack of frames, one per open list or map, each counting how many items were written so far. The struct fields are written by the type itself through begin_map, field and end_map; that is exactly what #[derive(Serialize)] generates.

Rust Serialize and Serializer with a JSON format

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Look at what the output tells you. The nested server map and the lists come out in exactly the order the fields were declared. A None became null, and a Some was written as just its inner value (JSON has no marker for it). The quote and newline inside the strings were escaped. The type never mentioned JSON; Config and Server only used the data model.

A second format: binary, and why it needs a schema

Now the payoff. A second Serializer writes compact binary: booleans as one byte, integers as eight little-endian bytes, strings as a four-byte length followed by UTF-8, lists as a length followed by the items, and Option as a tag byte (0 or 1). It ignores field names completely, because the order of fields in the struct is the layout.

This is the key difference between two kinds of formats:

  • Self-describing formats (JSON, TOML, YAML) carry names and structure in the data. A reader can inspect a document without knowing what it contains.
  • Positional formats (bincode, many network protocols) store only the values. They are smaller and faster, but the reader must know the exact type in advance. Reading Config bytes as a different struct gives garbage or an error.

Reading is the mirror image of writing. A Deserialize trait with a Reader over a byte slice builds a value. Everything that comes from outside is untrusted, so every step is checked: running out of bytes, a boolean byte that is neither 0 nor 1, invalid UTF-8, a number that does not fit the target type, and leftover bytes at the end. The reader also never uses a length read from the data to pre-allocate memory, since a hostile input could claim to contain four billion items.

To avoid repeating the boilerplate for each struct, the next example defines a record! macro that generates the struct and both trait implementations from a field list. That is a small stand-in for derive; the real one is a procedural macro (see Procedural Macros) but it produces code of the same shape.

Rust A JSON writer, a binary format and a derive stand-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The same Serialize implementations produced both outputs, and the same macro produced the reading code. Check the sizes: the binary form takes 81 bytes against 118 for the JSON text, because it does not repeat names such as host and port. In exchange, it cannot be read without the Config type, and it breaks the moment you reorder or add a field, unless you add a version number.

What the formats look like

Here is the same kind of configuration in the three families, so you can compare how each handles structure. The first two are text, the third is the byte layout from above.

JSON
{"name":"demo","server":{"host":"localhost","port":8080,"tls":false,"tags":["a","b"],"timeout":1.5},"retries":[1,2,3]}
TOML
name = "demo"
retries = [1, 2, 3]

[server]
host = "localhost"
port = 8080
tls = false
tags = ["a", "b"]
timeout = 1.5

TOML is designed for configuration files written by humans: comments, no required quotes around keys, and nested structs become [tables]. It also has a rule that complicates streaming writers: plain values must come before sub-tables, which is why a TOML serializer often has to buffer. That is one reason the data model lets a format reorder or buffer what it writes. Binary formats have none of these concerns, but nothing in them is readable by eye.

Property JSON TOML Binary (positional)
Human-readable yes yes, with comments no
Self-describing yes yes no
Missing values null key left out tag byte
Typical use web APIs config files caches, protocols
Reader needs the type no no yes

The same thing with real serde

With the actual crates, the whole lesson collapses to this. These snippets are read-only because the editor has no access to crates:

TOML
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "0.8"
Rust
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct Server {
    host: String,
    port: u16,
    #[serde(default)]
    tls: bool,
    tags: Vec<String>,
    #[serde(skip_serializing_if = "Option::is_none"Option::is_none")]
    timeout: Option<f64>,
}

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct Config {
    name: String,
    server: Server,
    retries: Vec<u32>,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let text = std::fs::read_to_string("config.toml")?;
    let config: Config = toml::from_str(&text)?;        // TOML in
    println!("{}", serde_json::to_string_pretty(&config)?); // JSON out
    Ok(())
}

The derive generates the equivalent of our record! expansion, but for both traits and with many options. The attributes you will use most are #[serde(rename = "...")] for a different external name, #[serde(rename_all = "camelCase")], #[serde(default)] for fields missing in the input, #[serde(skip_serializing_if = "...")], #[serde(flatten)] to merge a nested struct into its parent, and #[serde(tag = "type")] to control how enums appear in JSON.

On the reading side, real serde does not build an intermediate tree. A Deserializer calls into a Visitor supplied by the type, which receives the fields as they stream past, so large documents are parsed straight into your structs. That is more involved than our positional reader, but the contract is the same: the format knows the syntax, the type knows the shape.

Practical lessons from the exercise

  • Describe the shape once. Keep the data model small. Every extra concept is something all formats must support.
  • Treat input as hostile. Validate lengths, ranges and encodings, and return errors with enough context to find the problem.
  • Plan for change. Self-describing formats tolerate added fields. Positional formats need an explicit version byte or schema.
  • Pick the format for the reader. People edit TOML, programs exchange JSON, hot paths use binary.
  • Floats are special. JSON cannot represent NaN or infinity, which is why our writer emits null; real serializers either error or follow a documented convention.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does the intermediate data model solve?

    Types implement Serialize once, formats implement Serializer once, and any combination works because both sides speak the data model.
  2. 2. Why can a positional binary format not be read without knowing the target type?

    Self-describing formats include names and structure. A positional format contains just the values in order, so the reader's type decides how to interpret each byte.
  3. 3. Why does the Vec reader use Vec::with_capacity(len.min(1024)) instead of with_capacity(len)?

    The reader limits the up-front allocation. If the length is a lie, the loop fails with an end-of-input error long before memory is exhausted.
  4. 4. What does #[derive(Serialize)] generate, conceptually?

    The derive writes the same kind of code as the hand-written impl in this lesson, but format-agnostic and with support for attributes.

Summary

  • Serialization separates what a type is (Serialize, written in terms of a small data model) from how a format writes it (Serializer).
  • With that split, M types and N formats need M + N implementations instead of M x N.
  • begin_map, field and end_map is what #[derive(Serialize)] produces; a macro_rules! macro can imitate it.
  • JSON and TOML are self-describing; a positional binary format is smaller but needs the schema and a versioning plan.
  • Decoding must treat all input as hostile: check lengths, tags, UTF-8 and ranges, and never size allocations from untrusted numbers.
  • In real projects use serde with serde_json, toml or a binary crate, and keep these ideas in mind when debugging.

Next: Testing in Rust