Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Projects and Practice

Project: A Reproducible Experiment Tracker

Build a lightweight experiment log for notebook work: record parameters, seed, versions, data fingerprint and metrics per run, compare runs and reproduce the best one.

Anyone who has tuned a model in a notebook knows the pain: you changed three settings, got a better score, and an hour later cannot remember which three. Tools like MLflow and Weights & Biases solve this at scale, but the core idea is small: for every run, write down exactly what went in and what came out, in a file you can query later.

In this project you build that log yourself in plain Python, using ideas from this chapter: parameters and seeds (seeds and determinism), environment versions (reproducible environments), and a fingerprint of the data. Everything runs in the sandbox with a tiny, fully deterministic "model", so the numbers are stable. In your own work you replace the model function with real training code.

What a run record contains

flowchart LR
  A["Parameters"] --> R["Run record"]
  B["Seed"] --> R
  C["Data fingerprint"] --> R
  D["Environment versions"] --> R
  E["Metrics"] --> R
  R --> F["runs.jsonl"]

A record is only useful if it holds everything needed to repeat the run.

We store one JSON object per line in a runs.jsonl file (JSON Lines), which is easy to append to, diff and load into pandas with pd.read_json("runs.jsonl", lines=True).

Step 1: a deterministic toy experiment

Our experiment fits a straight line to noisy data by simple gradient descent. The seed controls the noise; the learning rate and the number of steps are parameters; the metric is the final mean squared error.

Python Step 1: the experiment function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The repeat is identical because every source of randomness is seeded through a local Random object. If you had used the global random.random() and forgotten the seed, the comparison would print False.

Step 2: fingerprint the data

A result only means something for a particular dataset. A fingerprint is a short hash of the data; if it changes, you know the input changed.

Python Step 2: hash the data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Equal data gives equal fingerprints and a tiny change gives a different one. For files use hashlib.sha256(open(path, "rb").read()) (or read in chunks for big files).

Step 3: the tracker

The tracker wraps a run: it records parameters and seed, calls the experiment, and appends a record. The store is a list here; on your machine it is a file opened in append mode.

Python Step 3: record runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each record is one JSON line. The Python line prints only the major.minor version, so the lines are stable between patch releases.

Step 4: compare and pick the best

Now query the log: sort by the metric, filter by name, and show a table.

Python Step 4: query the log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sorted table shows that more steps and a larger learning rate give a lower error in this tiny problem. The "best" record has everything needed to reproduce it: the parameters and the seed.

Step 5: reproduce a recorded run

The point of the log: pick any record and recreate its result exactly. Add a check that the recomputed metric equals the stored one; if not, something in the environment or data changed.

Python Step 5: reproduce and verify

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Taking it into a real notebook

  • Keep make_data, train and the tracker in a module (see moving code into modules), and keep the notebook for the narrative and charts.
  • Add numpy, pandas and scikit-learn versions to the record with importlib.metadata.version.
  • Add the Git commit hash: subprocess.run(["git", "rev-parse", "--short", "HEAD"], capture_output=True, text=True).stdout.strip(), and a flag if the working tree is dirty. A result from uncommitted code cannot be reproduced.
  • Load the log into a DataFrame and sort or plot it.
  • When the team grows, switch to MLflow, Weights & Biases or DVC; they store the same fields with a web interface and artifact storage, so your habit carries over.

Extensions

  • Add random search: draw lr and steps from ranges with a seeded generator and record all trials.
  • Store plots (saved as files) and refer to their paths in the record.
  • Write a compare(run_a, run_b) function that prints which fields differ.
  • Add the data fingerprint to the check in step 5, so that changed data raises a clear error.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must a run record contain to make a result reproducible?

    Everything that influenced the result has to be written down along with the result.
  2. 2. What is a data fingerprint?

    Hashing the data identifies exactly which input a result belongs to.
  3. 3. Why is JSON Lines (one JSON object per line) convenient for logs?

    Appending a line is cheap and safe, and read_json(lines=True) loads the whole log.
  4. 4. A reproduced run gives a different metric than the stored one. What might have changed?

    Differences point to something outside the record, which is exactly what the check is for.

Summary

  • Write down parameters, seed, data fingerprint, versions and metrics for every run.
  • Append records as JSON Lines, then query, sort and compare them.
  • Verify reproducibility by recomputing a stored run from its record.
  • Add the Git commit and move to MLflow or similar when you need a shared service.

Next: Practice Questions and Challenges