Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 76 ยท Reproducibility and Production Habits

Reproducible Analysis

Make analyses repeatable: project layout, random seeds, pinned environments, data fingerprints, config files, run manifests, and notebooks versus scripts.

Six months from now someone asks, "Which data and settings produced the number in that report?" That someone may be a colleague, an auditor, or you. If the only honest answer is "I think I ran the notebook, but I changed some cells afterwards", the result cannot be trusted, defended or improved. Reproducibility means that another person (or you, later, on another machine) can run your project and get the same numbers.

It is not a luxury for large teams. Reproducible work is also faster work: you can rerun an analysis when data is corrected, compare an experiment with last month's, and hand the project over without a long call. This lesson covers five practical habits: a clear project layout, controlled randomness, a pinned environment, fingerprinted data, and recorded configuration. It ends with the great notebook-versus-script debate.

What can differ between two runs

When a result changes, one of four things changed:

flowchart LR
    R["Result"] --> C["Code"]
    R --> D["Data"]
    R --> E["Environment<br/>(library versions)"]
    R --> S["Randomness and settings<br/>(seeds, parameters)"]

A result is reproducible only when code, data, environment and settings are all under control.

The rest of the lesson takes these in turn. Keep the idea of a run record: a small file saved with every important result that says which code, data, environment and settings produced it.

A project layout

A predictable folder structure matters more than the particular choice. This one works for most analyses:

TEXT
churn-analysis/
โ”œโ”€โ”€ README.md              what the project is, how to run it
โ”œโ”€โ”€ pyproject.toml         or requirements.txt: pinned dependencies
โ”œโ”€โ”€ config/
โ”‚   โ””โ”€โ”€ base.yaml          every setting that can change a result
โ”œโ”€โ”€ data/
โ”‚   โ”œโ”€โ”€ raw/               never edited, never committed if large
โ”‚   โ”œโ”€โ”€ interim/           cleaned intermediate tables
โ”‚   โ””โ”€โ”€ processed/         final modelling tables
โ”œโ”€โ”€ notebooks/             exploration and reporting only
โ”œโ”€โ”€ src/churn/
โ”‚   โ”œโ”€โ”€ load.py            read raw files, check the schema
โ”‚   โ”œโ”€โ”€ features.py        build model inputs
โ”‚   โ”œโ”€โ”€ train.py           fit and evaluate
โ”‚   โ””โ”€โ”€ report.py          tables and figures for the write-up
โ”œโ”€โ”€ tests/                 checks for the functions in src/
โ”œโ”€โ”€ outputs/               figures, tables, models, run records
โ””โ”€โ”€ Makefile               or run.sh: one command to rebuild everything

The principles behind it:

  • Raw data is read-only. All changes happen in code, so you can always rerun from the original and explain every step.
  • Code that matters lives in functions in src/, not in cells. Functions can be tested, reused and imported by both notebooks and scripts.
  • Outputs are generated, never edited by hand. If a figure needs a change, change the code that makes it.
  • One command rebuilds everything, so "how do I run this?" has a one-line answer.
Terminal
# rebuild every table, model and figure from the raw data
python -m churn.load config/base.yaml
python -m churn.features config/base.yaml
python -m churn.train config/base.yaml
python -m churn.report config/base.yaml

Version control (Git) records every change to code and config. Large data files usually stay out of Git; they are stored in a shared drive, a bucket or a data-versioning tool, and the project records a fingerprint of the file instead (see below). To start with Git, see the Git tutorial.

Controlling randomness

Many steps in a data project are random: shuffling rows, splitting into train and test sets, initializing models, bootstrapping, sampling. A seed is a number that fixes the sequence of pseudo-random numbers, so the same code produces the same "random" choices every time.

Good seed hygiene:

  • Create one random generator at the top of the script (np.random.default_rng(seed)) and pass it to the functions that need randomness, instead of calling a hidden global generator.
  • Pass random_state= to every scikit-learn splitter and estimator that accepts it.
  • Put the seed in the config file, not in the middle of the code.
  • Use independent streams for independent tasks, with SeedSequence.spawn, so adding a new random step in one place does not shift the numbers in another.
Python Seeds make randomness repeatable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Block 3 is the trap: with a single shared generator, deleting or adding one random call shifts every number after it. Spawned streams (block 4) avoid the problem, because each task owns its own sequence.

A seed gives reproducibility, not truth. A result that only looks good with seed 42 is luck. For important comparisons, repeat with several seeds and report the average and the spread.

Some sources of randomness are harder to pin: multi-threaded training can change the order of floating-point additions, and some libraries use hardware-specific code. Reproduce to within a small tolerance and say so, rather than promising bit-for-bit identity.

Pinning the environment

Library versions change results: a default parameter changes, a bug is fixed, a function is deprecated. Record exactly what you used:

TEXT
# requirements.txt (pinned)
numpy==2.4.6
pandas==3.0.2
scikit-learn==1.8.0
matplotlib==3.10.8

Create one virtual environment per project and install from the pinned file (see virtual environments). Lock files (uv.lock, poetry.lock, pip-tools output) also pin the dependencies of your dependencies. For the highest level of control, a Docker image freezes the whole system, including the Python version and operating-system libraries. The run record (below) should also store the versions that were actually imported, because the file on disk and the running environment can disagree.

Fingerprinting data

How do you know the data of today's run is the data of last month's? Compare a hash, a short fixed-size fingerprint of the file's bytes. If one byte changes, the fingerprint changes completely. Save the hash with every result. Tools such as DVC and lakeFS manage versions of big files this way, and databases offer snapshots and time travel, but the principle is the same: name the exact data.

Python A fingerprint for a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For large files you hash the file's bytes in chunks (hashlib.sha256() with update) instead of building a CSV in memory. Whatever method you use, keep it fixed across the project.

Configuration in one place

Hard-coded numbers scattered through a script are invisible settings. Gather everything that can change the result (paths, seed, split size, model parameters, thresholds) in a config file such as YAML or JSON, committed to Git. Then a different experiment means a different config, not edited code, and the config is part of the run record.

TEXT
# config/base.yaml
seed: 42
data:
  path: data/raw/orders.csv
  test_size: 0.25
model:
  name: logistic_regression
  C: 1.0
  max_iter: 500

Keep secrets (passwords, API keys) out of config files and out of Git; read them from environment variables.

A run record in action

The example below brings the pieces together in a form you can copy: a config (JSON, because YAML needs an extra package), a pure function that does the work, and a run record with the config, seed, data fingerprint, library versions and metrics. Running it twice gives identical records, except for fields that are meant to differ (we leave out the timestamp from the comparison).

Python A config-driven run with a run record

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real project you would write the record to outputs/run-<id>.json next to the figures and tables it describes, and commit or archive it. The last lines show the point of the exercise: a different seed gives a different dataset, and the fingerprint and the record make that difference visible instead of mysterious.

Notebooks versus scripts

Jupyter notebooks are excellent for exploring, explaining and presenting. They are weak at reproducibility, for three reasons: cells can run in any order (the state on screen may not match the code that produced it), they diff badly in Git, and logic buried in cells is hard to test.

A sensible division of labour:

Use a notebook for Use a script or module for
Exploring data and trying ideas Loading, cleaning and feature code you will run again
Narrating the analysis with text and charts Training and evaluation
Presenting results Anything a scheduler or another person must run

The workflow that scales: explore in a notebook, move the code that worked into functions in src/, and have the notebook import them. Before sharing a notebook, restart the kernel and run all cells from the top; if it fails or gives different numbers, it was not reproducible. Tools such as nbstripout (removes outputs before committing) and papermill (runs notebooks with parameters) help; the project workflow lesson discusses keeping projects script-friendly.

A reproducibility checklist

  • Raw data is untouched; every change is in code.
  • One command rebuilds the results from the raw data.
  • Seeds are in the config and passed to every random step.
  • Dependencies are pinned, and the versions used are recorded.
  • The data has a fingerprint or version, stored with the results.
  • Settings live in a config file under version control.
  • A fresh clone on another machine produces the same numbers (try it!).

The last line is the real test. If you have never rebuilt a project from scratch on a clean machine, you do not yet know whether it is reproducible.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the purpose of setting a random seed?

    A seed fixes the pseudo-random sequence, so splits, shuffles and sampling give the same result every run. It does not change accuracy by itself.
  2. 2. Why is it risky to share one random generator across many steps?

    The sequence advances with each call. Independent streams (SeedSequence.spawn) keep steps from affecting each other.
  3. 3. What does a data fingerprint (hash) let you check?

    Any change to the content changes the hash, so storing it with a result names the exact data version.
  4. 4. Which is the best home for the train/test split size and the model parameters?

    A config makes all settings visible, comparable between experiments and part of the run record.
  5. 5. What is the quickest test that a notebook is reproducible?

    Out-of-order cells can leave hidden state. A clean top-to-bottom run exposes the problem.

Summary

  • A result depends on code, data, environment and settings; a reproducible project controls all four.
  • Use a predictable layout: read-only raw data, reusable code in src/, generated outputs, one command to rebuild.
  • Create generators from a seed in the config, pass random_state everywhere, and use independent streams for independent steps.
  • Pin dependencies and record the versions actually used; per-project environments or containers help.
  • Fingerprint the data and save the hash with every result.
  • Keep settings in a config file, secrets in environment variables, and write a run record for important results.
  • Explore in notebooks, keep real logic in scripts and functions, and test reproducibility by rebuilding from a clean state.

Next: Testing Data Code and Data Validation, to catch mistakes before they reach your results.