Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 41 ยท Data and Experiment Tracking

Comparing Experiments and Tuning

Learn how to compare model runs fairly with cross-validation and seeds, run grid and random hyperparameter searches, and log every trial so the best model can be justified.

Once you track runs, the next question is which one is better. A difference of 0.5 percentage points might be a real improvement, or it might be luck from a different random split. This lesson covers fair comparison and systematic hyperparameter tuning, with every trial logged like the runs in experiment tracking.

Compare fairly

Two runs are comparable only if you hold everything else constant. Check this list before you trust a difference:

  • Same data and same split. Different train/test splits alone can move accuracy by a point or more.
  • Same metric, computed the same way, on a set the model never trained on.
  • Repeated measurement. Use cross-validation or several seeds, and look at the spread, not only the mean.
  • One change at a time, or you will not know which change helped.
Python Mean and spread across folds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the means differ by less than the standard deviation, treat the models as tied and prefer the simpler or cheaper one.

Hyperparameters are settings chosen before training. Two common strategies:

Strategy How it works Best when
Grid search Try every combination from small lists. Few parameters, cheap training.
Random search Sample combinations at random from ranges. Many parameters; some matter far more than others.

Random search often finds good settings faster, because with a grid you repeat the same few values of the important parameter, while random sampling tries a new value every time.

Python Grid search with logged trials

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the three sets of data: tuning used cross-validation inside the training set, and the held-out test set was touched once at the end. That test number is your honest estimate. cv_results_ is already a table of trials, ready to log.

Python Random search over ranges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Scale-like parameters (learning rates, regularization strengths) are searched on a log scale, because 0.001 versus 0.01 matters as much as 1 versus 10.

Beyond grid and random

Larger projects use Bayesian optimization (Optuna, Hyperopt) that learns from earlier trials where to look next, and early stopping or successive halving to abandon poor trials cheaply. They plug into the same tracker: each trial is a run with parameters and a metric.

Every trial you try is another chance to get lucky. The more settings you compare, the more the best validation score overestimates real performance, which is why a final untouched test set matters.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use the same cross-validation folds when comparing two models?

    Using identical folds removes split luck as an explanation for a score difference.
  2. 2. When is random search usually better than grid search?

    Random search tries many distinct values of every parameter, so it explores the important ones better than a grid that repeats the same few values.
  3. 3. Why search learning rates on a log scale?

    Scale parameters act multiplicatively, so sampling uniformly in log space covers orders of magnitude evenly.
  4. 4. What is the role of the held-out test set after tuning?

    The test set is used once at the end; using it for choices makes the estimate optimistic.

Summary

  • Compare runs on identical data and splits, with repeated measurements and a spread.
  • Grid search tries all combinations; random search samples ranges and scales better.
  • Search scale-like parameters on a log scale.
  • Log every trial (cv_results_ is already a table) and keep the final test set untouched.
  • Smarter tools (Optuna, successive halving) use the same run-and-metric structure.

Next: Saving and Loading Models