Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 41 ยท Data and Experiment Tracking

Experiment Tracking Basics

Learn what an experiment run records, build a small tracker in plain Python, compare runs, and see how tools like MLflow and Weights and Biases structure the same idea.

You train a model, get an accuracy of 0.91, change a setting, get 0.93, and a week later nobody remembers which settings produced which number. Experiment tracking fixes this by recording every training attempt as a run with everything needed to understand and repeat it.

A tracker is the memory of your project. Without it, results live in notebook cells and chat messages; with it, any result can be found, compared and reproduced.

What a run should record

One run, four records

Identity Run id, name, time, author.
Parameters Every setting you chose.
Metrics Scores, losses, timings.
Versions Code commit, data hash.

Plus artifacts - the model, plots, reports.

1 / 2

The record has a clear split. Parameters are inputs you control (learning rate, number of trees). Metrics are outputs you measure (accuracy, F1). Keep them apart: you compare runs by asking "which parameters gave the best metric".

A tracker in thirty lines

Real tools are bigger, but the core is small. This one appends each run to a list; a real tracker writes to disk or a server.

Python A tiny experiment tracker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the seeds are part of the parameters. Without a recorded seed, a run cannot be repeated.

Querying runs

Once runs are structured data, "which was best?" becomes a table operation. Flatten the nested dictionaries into columns.

Python Compare runs in a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What real tools add

Tools such as MLflow, Weights and Biases, Neptune and Comet wrap this idea with extras:

  • Logging calls inside your training code, for example log_param, log_metric and log_artifact in MLflow.
  • Metric history: a value per step or epoch, so you can plot learning curves.
  • Artifact storage: models, plots and reports attached to a run.
  • A web UI to sort, filter and chart runs, and an API to query them.
  • Auto-logging: hooks that capture scikit-learn or PyTorch settings and metrics automatically.

The shape of an MLflow script is about this (read-only here, because the MLflow package and server are not part of this page's runner):

Python
import mlflow

with mlflow.start_run(run_name="rf-depth8"):
    mlflow.log_params({"n_estimators": 50, "max_depth": 8})
    model.fit(X_train, y_train)
    mlflow.log_metric("accuracy", accuracy)
    mlflow.sklearn.log_model(model, "model")

The concepts (run, parameter, metric, artifact) are identical to our tiny tracker.

Habits that make tracking useful

  1. Log everything automatically, not just numbers you remember to type.
  2. Record the code version (a Git commit hash) and the data fingerprint with every run.
  3. Name runs and use tags (stage=baseline, ticket=1423) so you can filter later.
  4. Log the merged configuration, not only the file name.
  5. Keep failed runs. Knowing that an idea did not work stops you from trying it again.

Do not tune on the test set and then report the test score as an honest estimate. Log validation metrics for decisions and report test metrics once, at the end.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between a parameter and a metric in a run?

    Parameters (learning rate, depth) are inputs to a run; metrics (accuracy, loss) are its measured outputs.
  2. 2. Why record the random seed as a parameter?

    The seed controls randomness; recording it is part of making a run reproducible.
  3. 3. Which item is an artifact of a run?

    Artifacts are files produced by a run, such as models, plots and reports.
  4. 4. Why keep failed or poor runs in the tracker?

    Negative results are knowledge. Keeping them prevents repeated work.

Summary

  • A run records identity, parameters, metrics, versions (code and data) and artifacts.
  • Keep parameters and metrics separate so runs can be compared by a simple table query.
  • Tools like MLflow, Weights and Biases, Neptune and Comet add logging calls, UI, storage and auto-logging around the same idea.
  • Track the seed, code commit and data fingerprint, and keep failed runs.

Next: Comparing Experiments and Tuning