Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 73 ยท Frameworks and Production

Experiments and Reproducibility

Run deep-learning experiments you can trust: fix seeds, store configs, log metrics, compare random and grid search, and version your data and models.

A single training run proves very little. If you change the learning rate and the accuracy goes up by one point, was it the learning rate, or just a luckier random initialization? A week later, can you even say which code, data and settings produced the model you are about to ship? Deep-learning work is full of randomness and full of knobs, so without discipline it quickly turns into a pile of numbers nobody can explain.

This lesson gives you a small set of habits that prevents that: control randomness with seeds, describe every run with a config, log metrics as you go, search hyperparameters sensibly, and version the data and the models. All of the examples run in the browser with the same NumPy networks you built earlier.

What makes a run reproducible

Seeds Fix every source of randomness you control.
Config Every setting lives in one saved file.
Data version Know exactly which data you trained on.
Code version Record the commit that ran.

Same seed, config, data and code should give the same numbers.

1 / 2

Seeds: taming the randomness

Random numbers enter a training run in several places: the initial weights, the shuffling of the data, dropout masks, data augmentation, and the split into training and validation sets. A seed fixes the starting point of a random number generator, so the same seed produces the same sequence every time.

In NumPy the modern way is a dedicated generator object, rng = np.random.default_rng(seed), which you pass around instead of relying on hidden global state. In PyTorch you set several seeds at once:

Python
import random, numpy as np, torch

def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)          # CPU and, via it, the GPU generators

Setting seeds is necessary but not always sufficient. Some GPU operations are non-deterministic for speed, data-loader workers each need their own seed, and different library versions or hardware can change floating-point results slightly. For strict reproducibility you also record the library versions, and in PyTorch you can ask for deterministic algorithms (at some cost in speed). The practical goal is usually "close enough to compare", not "bit for bit".

Seeds also expose something important: results vary from seed to seed. A difference smaller than that variation is noise. The first example shows both ideas. It defines a config, trains a small network, proves that the same seed gives identical results, and measures the spread over several seeds.

Python Seeds, a config and run logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same seed gives exactly the same log, and a different seed does not. Over ten seeds the validation accuracy averages 0.818 with a standard deviation of 0.042, anywhere from 0.75 to 0.89 (the split changes with the seed, and the validation set has only 100 points). Look at the spread across seeds before you celebrate a gain: if two settings differ by less than that spread, the experiment has not shown a real difference. Report the mean over several seeds, not the best single run.

Configs instead of edited code

The settings of a run (learning rate, layer sizes, epochs, data path, augmentation, seed) should be data, not constants scattered through a script. Put them in one object or file, usually YAML or JSON, and save a copy next to every result:

YAML
seed: 0
model: {hidden: 16, dropout: 0.1}
optimizer: {name: adamw, lr: 0.001, weight_decay: 0.01}
data: {path: data/v3, batch_size: 64}
epochs: 80

Two benefits follow. You can rerun any experiment from its saved config, and you can compare two runs by comparing two small files instead of reading two versions of the code. The example above also hashes the config into a short run id, a simple way to name a run so that its logs, plots and weights can be found later. Tools such as Hydra or plain argparse plus a YAML file load configs and allow command-line overrides.

Logging metrics

Log at least the training loss, the validation loss and the key metric (see Evaluating Models) after every epoch, plus the learning rate and the time taken. The simplest format is JSON Lines: one JSON object per line, as printed above, which is easy to append to, easy to parse and easy to plot. Dedicated tools add dashboards and comparison tables on top of the same idea: TensorBoard, MLflow, and Weights & Biases are common choices, and they all boil down to "record a dictionary of numbers against a step number".

Log more than the metric curves. Save the config, the git commit hash of the code, the library versions, the data version, and a handful of example predictions. The week you need them, you will not remember.

Hyperparameters are the settings you choose rather than learn: learning rate, layer sizes, batch size, dropout rate, weight decay. Finding good values by hand is slow, so we search.

  • Grid search tries every combination from lists you choose, for example 3 learning rates times 3 widths is 9 runs. The cost grows multiplicatively with each new hyperparameter.
  • Random search draws each setting independently from a range for a fixed number of trials. It is a better default because usually only one or two hyperparameters matter much (very often the learning rate), and a grid wastes runs repeating the same few values of the important one. Random search tries a new value of every hyperparameter in every trial.
  • Smarter methods such as Bayesian optimization and early-stopping schedulers (successive halving, Hyperband) also exist, and libraries like Optuna implement them. Random search is the baseline to beat.

Two practical rules: sample scale-like parameters (learning rate, weight decay) on a log scale, because the difference between 0.001 and 0.002 matters as much as between 0.1 and 0.2; and always judge trials on validation data. The test set stays untouched until the end, otherwise your choice overfits to it.

The next example compares the two strategies fairly. Both get a budget of 9 training runs on the same task. The grid uses three learning rates and three widths. Random search is repeated 20 times with different seeds, since one random search is itself a random event.

Python Grid search versus random search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scatter plot shows the geometry. The grid lines up three columns, so if the best learning rate falls between them, it is never tested. Random search scatters nine different learning rates, so it samples the important axis much more densely for the same budget.

The printed numbers are the honest part. On this tiny two-hyperparameter task the two strategies end up almost level: the grid's best loss is 0.288, the random searches average 0.287, and random search wins in 9 of 20 repeats. This grid happened to contain a good learning rate (0.3). Move the grid values and the result changes, while the random search keeps its average. The advantage of random search appears as you add hyperparameters: a grid over five settings with three values each needs 243 runs, yet only a few of those settings matter and each gets just three values. A single search, grid or random, is also noisy, so do not read much into one run of either.

Versioning data and models

Code lives in git, and the same care applies to everything else a result depends on:

  • Data: keep raw data read-only, never edit it in place, and record a version. A content hash (SHA-256 of the file bytes) is a cheap fingerprint; tools such as DVC track large files by hash outside git. Write down how the train, validation and test splits were made, ideally with a saved list of ids and not only a seed.
  • Models: save weights together with the config, the data version and the code commit. A model registry (MLflow and cloud services provide them) assigns versions and stages such as "staging" and "production" so you can roll back.
  • Environment: pin library versions in a lock file or a container image.

A quick demonstration of fingerprinting data, using only the standard library:

Python
import hashlib, numpy as np

X = np.arange(10, dtype=np.float32)
print(hashlib.sha256(X.tobytes()).hexdigest()[:12])      # changes if any value changes

Treat every run as a small scientific experiment: one question, one change, a saved config, several seeds, and a validation metric. Resist tuning against the test set, however tempting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two models differ by 0.4 percentage points of accuracy on one run each, while the same model varies by 1.5 points across seeds. What should you conclude?

    When the seed-to-seed spread is larger than the difference, a single run each proves nothing. Compare means over several seeds.
  2. 2. Why is random search usually preferred to grid search for the same budget?

    A grid repeats the same few values of each hyperparameter. When only one or two matter, random draws explore them far better.
  3. 3. How should a learning rate be sampled in a random search?

    Learning rates act multiplicatively, so a log scale spreads trials evenly across orders of magnitude.
  4. 4. Which set of things should be saved with a trained model so that the run can be reproduced?

    Weights alone do not say how they were produced. The other items let you rerun the experiment or audit it later.

Summary

  • Fix and record the seeds, but expect some variation anyway; compare settings using several seeds and report mean and spread.
  • Keep every setting in a config saved with the run, and give each run an id.
  • Log metrics each epoch in a simple format such as JSON Lines, plus code commit, data version and library versions.
  • Search hyperparameters on validation data; random search with log-scale sampling is a strong default and beats a grid for the same budget when few parameters matter.
  • Version data (hashes, splits) and models (weights with config) like code.

Next: Deploying and Serving Models.