Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 111 ยท Model Evaluation and Tuning

Hyperparameter Tuning

Tune models with GridSearchCV, RandomizedSearchCV and successive halving: build parameter grids, sample from distributions, read cv_results_ and stay within a time budget.

Every model has settings you choose before training: the C of an SVM, the max_depth of a tree, the alpha of a ridge regression. These are hyperparameters. The model never learns them from the data, so somebody has to pick them. You can guess, copy a value from a tutorial, or search for good values in a disciplined way. This lesson is about the third option.

The recipe is always the same: define a set of candidate settings, score each candidate with cross-validation, and keep the best. scikit-learn wraps that loop in a few search classes that behave like ordinary estimators, so you call fit and predict as usual. By the end you will know when to use a grid, when to sample randomly, and how to stop a search from eating your time budget.

Three ways to search

Grid search Try every combination you list.
Random search Sample a fixed number of combinations.
Halving Start cheap, keep only the winners.

All three score candidates with cross-validation on the training data only.

1 / 2

Hyperparameters versus parameters

A parameter is learned during fit: a coefficient, a split threshold. A hyperparameter is set in the constructor and stays fixed during fit. In scikit-learn the learned ones end with an underscore (coef_), and the hyperparameters show up in get_params() (see The scikit-learn API).

The golden rule for tuning is that the test set stays untouched. If you try 200 settings and pick the one with the best test score, the test score is no longer an honest estimate, because you chose the winner for being lucky on exactly that data. Tuning uses cross-validation inside the training set, and the test set is used once, at the very end.

GridSearchCV: try every combination

GridSearchCV takes an estimator and a dictionary mapping each hyperparameter name to a list of values to try. It fits and scores every combination on every fold. When the grid has 3 values of C and 3 values of gamma, that is 9 candidates; with 5 folds, 45 fits. Afterwards it refits the best candidate on the whole training set, so the search object itself is ready to predict.

When the estimator is a pipeline, parameter names use the step__parameter convention (two underscores). Putting the scaler inside the pipeline matters: each fold then fits the scaler on its own training portion only. More on that in Nested Cross-Validation and Data Leakage.

Python A grid search on an SVM pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details worth noticing. First, the table includes std_test_score: when two candidates differ by less than that spread, the difference is noise, not signal. The top two settings here tie on mean score, and rank_test_score shows 1 for both. best_params_ simply takes the first of the tied candidates. When ties happen, prefer the simpler or cheaper model. Second, the heat map reads rows as C and columns as gamma because that is the order in which GridSearchCV enumerates the grid (the last parameter varies fastest), which is why reshaping works.

Grid search cost multiplies: 4 hyperparameters with 5 values each is 625 candidates, times the number of folds. Grids explode quickly, which is the main reason random search exists.

Grids can be lists of grids

Some hyperparameters only make sense together. A penalty="l1" needs a solver that supports it, and a kernel="linear" SVM has no gamma. Pass a list of dictionaries and scikit-learn explores each dictionary as its own grid:

Python
param_grid = [
    {"kernel": ["linear"], "C": [0.1, 1, 10]},
    {"kernel": ["rbf"], "C": [0.1, 1, 10], "gamma": [0.01, 0.1, 1]},
]

That is 3 + 9 = 12 candidates instead of the 18 you would get from crossing everything and wasting fits on meaningless combinations.

RandomizedSearchCV: sample instead of enumerate

Random search draws a fixed number of candidates (n_iter) from distributions you describe. It has two big advantages. The cost is set by you, not by the size of the space. And continuous hyperparameters are explored properly: a grid of [0.1, 1, 10] tests exactly three values of C, while 15 random draws test 15 different ones. In practice, only a few hyperparameters matter much for any given dataset, and random search tries many more distinct values of those important ones than a grid with the same budget.

You describe the space with lists (sampled uniformly) or with scipy.stats distributions: randint(2, 12) for integers, uniform(0.1, 0.8) for a continuous range, and loguniform(1e-3, 1e2) for values whose scale matters. Parameters like C, alpha and learning rates behave multiplicatively, so a log-uniform draw spends as much effort between 0.001 and 0.01 as between 10 and 100. Set random_state to make the draws reproducible.

Python Random search with scipy distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first search, many values of C above about 0.08 give nearly identical scores: past that point the penalty is too weak to matter on this data. A flat plateau like that is useful information. It says the model is not sensitive to C there, so you can pick any value on it.

Reading cv_results_

cv_results_ is a dictionary of arrays that converts directly into a DataFrame, one row per candidate. The columns you will use most:

Column Meaning
params The full settings of the candidate, as a dictionary.
param_<name> One column per hyperparameter, convenient for sorting and plotting.
mean_test_score, std_test_score Average and spread of the validation score across folds.
rank_test_score 1 for the best candidate. Ties share a rank.
mean_fit_time, mean_score_time How long fitting and scoring took, in seconds.
split0_test_score, split1_... The score of each individual fold.

Add return_train_score=True to also get training scores. A large gap between train and validation score for the best candidate is a warning about overfitting. Use scoring= to optimize something other than accuracy ("f1", "roc_auc", "neg_mean_absolute_error"): the metric you tune is the metric you get, so choose it to match the real goal, as discussed in Classification Metrics. You can even pass a list of metrics with refit="f1" to track several while selecting by one.

A sensible tuning workflow

  1. Start with defaults and a baseline. Defaults are often near a good value, and you need that score to know whether tuning helped.
  2. Search coarsely first: a wide random search over log scales, then look at cv_results_ to see which hyperparameters matter and where the good region lies.
  3. Zoom in: a smaller grid or narrower distributions around the good region.
  4. Check the gain against the noise. If the tuned score beats the default by less than std_test_score, you probably gained nothing real.
  5. Evaluate once on the held-out test set.

Tuning has diminishing returns. A well-chosen model family and good features usually beat an exhaustive search on a poor setup.

Successive halving

Why spend a full fit on a candidate that is obviously bad? Successive halving gives every candidate a small slice of the data first, keeps the best fraction (1/factor), gives the survivors more data, and repeats until one slice uses all the data. HalvingGridSearchCV and HalvingRandomSearchCV implement this. They are still marked experimental, so you must import a switch module first.

Python Halving: more data for fewer candidates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The printout shows the schedule: 24 candidates start on a small subset, a third of them survive each round, and the last few are scored on all the data. Halving works best when a model trained on a small sample ranks candidates roughly like one trained on everything. That is true for many forests and boosted trees, less true when small samples change which setting is best.

Time budgets in the browser

The practice runner on this site runs Python inside your browser, on one core, with a 30 second limit. A few habits keep searches inside it:

  • Count the fits before you press Run: candidates times folds (and times n_iter for repeated schemes).
  • Use RandomizedSearchCV with a small n_iter, 3-fold CV, and small forests (20 to 50 trees).
  • Tune on a subsample first, then run the finished plan on your own machine with all the data.
  • On your own computer, set n_jobs=-1 to use all cores. In the browser there is only one, so leave it at the default.
  • Cheap hyperparameters first: scalers, regularization strengths and tree depth are fast to evaluate. Thousands of trees are not.

Common mistakes

  • Tuning on the test set. Always split first, and tune with cross-validation on the training part.
  • Scaling or selecting features outside the pipeline. The search then leaks information into every fold. Put every data-dependent step inside the pipeline passed to the search.
  • Linear ranges for scale-like parameters. Use loguniform or powers of ten for C, alpha and learning rates.
  • Over-trusting the winner. The best score among many noisy candidates is biased upward. The next lesson shows how to measure that bias.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A grid has 4 values of C, 5 values of gamma and you use 5-fold cross-validation. How many model fits does the search do before the final refit?

    There are 4 x 5 = 20 candidates, and each is fitted once per fold, so 20 x 5 = 100 fits. The refit on all training data comes on top of that.
  2. 2. Why is loguniform(1e-3, 1e2) usually better than uniform(0.001, 100) for sampling C?

    A uniform draw between 0.001 and 100 lands above 1 about 99% of the time, so small values are barely explored. Log-uniform gives each power of ten equal weight, which matches how C behaves.
  3. 3. When is RandomizedSearchCV preferable to GridSearchCV?

    Random search lets you set the budget directly and tests many distinct values of each continuous hyperparameter. A grid grows multiplicatively with every parameter you add.
  4. 4. Two candidates have mean CV scores of 0.913 and 0.912, each with a standard deviation of 0.03 across folds. What is the sensible conclusion?

    A difference far smaller than the fold-to-fold spread is not evidence of a real gap. Breaking the tie on simplicity or cost is a better use of the information.

Summary

  • Hyperparameters are set before training; a search picks them using cross-validation on the training data only.
  • GridSearchCV tries every combination; RandomizedSearchCV samples n_iter of them from lists or scipy.stats distributions.
  • Use step__parameter names for pipelines, and put preprocessing inside the pipeline.
  • best_params_, best_score_ and the refit best_estimator_ summarize the result; cv_results_ (as a DataFrame) shows every candidate, its spread and its rank.
  • Halving searches give small samples to many candidates and more data to the survivors.
  • In the browser, count fits first and keep searches small; the best score of a search is optimistic, which the next lesson quantifies.

Next: Nested Cross-Validation and Data Leakage.