Skip to content
Tutorials Center
My learning

Topics

Lesson 80 of 111 ยท Beyond the Basics

Experiment Tracking with MLflow

Learn why every ML run should log its parameters, metrics and artifacts, how MLflow tracking and the model registry work, and build a mini tracker in Python.

A few weeks into a project you will have trained dozens of models. Which one scored best? Which learning rate did it use? Was it trained on last month's data or this month's? Which file is the model that is actually running in production? If the answers live in file names like model_final_v2_REAL.pkl and your memory, you cannot reproduce your best result, and neither can your teammates.

Experiment tracking solves this by recording, for every training run, everything needed to understand and repeat it. MLflow is the most widely used open-source tool for it. This lesson explains what to log, how MLflow's tracking and model registry work, and then builds a tiny tracker you can run in the browser, so that the ideas are concrete. The real MLflow needs a server and files on disk, so its code is shown read-only.

What to record for every run

  • Parameters: the settings you chose (C, max_depth, learning rate, random seed, feature list).
  • Metrics: the scores you got (accuracy, F1, AUC, loss per epoch), on validation data and, at the very end, on the test set.
  • Data identity: where the data came from and which version (a file hash, a dataset version, the date range).
  • Code identity: the Git commit hash of the code that produced the run.
  • Environment: library versions (scikit-learn==1.8), Python version.
  • Artifacts: the files the run produced: the trained model, plots, the confusion matrix, a feature list.
  • Tags and notes: who ran it, why, and what you changed.

With these, "reproducible" stops being a hope and becomes a query.

MLflow in four parts

  1. Tracking: an API and a UI for runs, parameters, metrics and artifacts. Runs are grouped into experiments.
  2. Models: a standard packaging format (MLmodel file plus the model and its environment), so the same model can be served with different tools.
  3. Model registry: a central catalogue of named models with numbered versions and labels such as champion or staging, so that production code can ask for "the current champion" without knowing a file path.
  4. Projects and deployment tools: packaging code and serving models (for example mlflow models serve).

Tracking in code

This is how the usual flow looks with the real library. It runs on your own machine after pip install mlflow, so it is read-only here:

Python
# read-only: needs the mlflow package and a place to store runs
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score

mlflow.set_tracking_uri("http://localhost:5000")      # or a local folder; omit to use ./mlruns
mlflow.set_experiment("churn-model")

params = {"n_estimators": 150, "max_depth": 12, "random_state": 0}
with mlflow.start_run(run_name="forest-depth12"):
    mlflow.log_params(params)
    model = RandomForestClassifier(**params).fit(X_train, y_train)
    mlflow.log_metric("f1_valid", f1_score(y_valid, model.predict(X_valid)))
    mlflow.set_tag("data_version", "2026-01-snapshot")
    mlflow.sklearn.log_model(model, name="model")    # saves the model and its environment as an artifact

For common libraries, mlflow.autolog() records parameters, metrics and the model automatically without any log_* calls. You then start the web interface with mlflow ui and compare runs in a table, sort by a metric and plot runs against each other.

The model registry

When a run produces a model worth keeping, you register it under a name. Each registration creates a new numbered version, and you point an alias at the version that should serve traffic:

Python
# read-only: registering and loading a model
import mlflow
from mlflow import MlflowClient

result = mlflow.register_model(f"runs:/{run_id}/model", "churn-classifier")      # creates a new version
MlflowClient().set_registered_model_alias("churn-classifier", "champion", result.version)

# in the serving code: always load whatever "champion" currently points to
model = mlflow.sklearn.load_model("models:/churn-classifier@champion")

Promoting a better model is then a one-line change of the alias, and rolling back is changing it back, with no redeploy of the serving code. (Older MLflow versions used fixed stages such as "Staging" and "Production" for this; aliases are the newer, more flexible replacement. Check the documentation of the version you install.)

A mini tracker in plain Python

To see what a tracker does, here is a small in-memory version with the same shape: start a run, log parameters and metrics, then query all runs. It is a teaching stand-in, not MLflow, and it forgets everything when the page is closed. We train four models on an imbalanced classification problem and compare them.

Python Log four runs and pick the best one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a table like this you can answer questions that would otherwise be guesswork: the forest with depth 12 gives the best AUC and F1, the logistic regressions barely change between C = 0.1 and C = 10, and the ranking by accuracy would have hidden how much better the forest handles the minority class (compare the F1 values, 0.65 to 0.66 against 0.75). Notice that we compared runs on a validation set; touch the test set only once, for the model you finally choose (nested CV and leakage).

Good habits

  • Name experiments after the question, not the date: churn-model, price-forecast.
  • Set and log all random seeds, and log the data version. A run you cannot reproduce is only a story.
  • Log failures too. Runs that did not work are information.
  • Keep the registry small and meaningful: register only models you might deploy, and write a short description with each version.
  • Automate it. Tracking should happen in the training script, not by hand. That is what makes CI/CD for ML possible.
  • Alternatives: Weights & Biases, Neptune, Comet, DVC (data and pipelines versioned with Git), or the tracking built into cloud platforms. The ideas are the same.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these should a good experiment record include?

    Reproducing a run needs the settings, the data, the code and the environment, plus the results and outputs.
  2. 2. What does the MLflow model registry add on top of tracking?

    The registry decouples what to serve (a name and an alias) from where the model files are stored.
  3. 3. Why should you compare runs on a validation set and not on the test set?

    Choosing among many runs is itself a fitting step. Keep the test set for a single final estimate.
  4. 4. Why is the in-page tracker in this lesson only a teaching tool?

    A real tracker persists everything, stores artifacts and offers a shared interface. The stand-in only shows the logging pattern.

Summary

  • Experiment tracking records parameters, metrics, data and code versions, environment and artifacts for every run so results can be compared and reproduced.
  • MLflow offers tracking, a model packaging format, a registry with versions and aliases, and serving tools; mlflow.autolog() logs common libraries automatically.
  • The registry lets serving code load "the champion" by name, so promotion and rollback do not need a code change.
  • Compare runs on validation data, log seeds and data versions, and automate tracking in the training script.

Next: let the machine run the whole pipeline on every change, in CI/CD for Machine Learning.