Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 40 ยท Model Evaluation and Selection

Hyperparameter Tuning with Grid and Random Search

Tune scikit-learn models with GridSearchCV, RandomizedSearchCV and successive halving; read cv_results_, search pipelines safely and avoid optimistic scores.

Parameters are learned from data (the coefficients of a linear model). Hyperparameters are settings you choose before training: the number of trees, the strength of regularization, the depth limit. The right values depend on the dataset, and defaults are only a starting point.

Hyperparameter search automates the choice: try many combinations, score each one with cross-validation, keep the best. scikit-learn offers exhaustive search (GridSearchCV), random sampling (RandomizedSearchCV) and faster successive-halving variants. All of them follow the same estimator API, so a fitted search object can be used like a model.

GridSearchCV

You give a param_grid: a dictionary of names to lists of candidate values. GridSearchCV tries every combination, cross-validates each, and refits the best one on all the data you passed.

Python GridSearchCV on a support vector machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notes on the code:

  • Parameters of a step in a pipeline are written stepname__parametername (double underscore).
  • 3 x 3 x 2 = 18 combinations, times 5 folds, is 90 fits. Grids grow multiplicatively, so they get expensive quickly.
  • After fitting, search.best_estimator_ is the refitted best model and search.predict uses it.
  • The test score is measured on data the search never saw. best_score_ is optimistic, because it is the maximum over many noisy estimates.

Reading the results

cv_results_ has a row per combination. A DataFrame makes it readable.

Python Inspecting cv_results_

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at std_test_score: if the top five candidates are within one standard deviation of each other, they are statistically tied, and you should prefer the simpler or faster one. A big gap between mean_train_score and mean_test_score is a sign of overfitting.

RandomizedSearchCV

When there are many hyperparameters, or continuous ones, a grid wastes time on unimportant combinations. Random search draws a fixed number of settings (n_iter) from distributions. Research on search strategies shows it often finds equally good settings with far fewer fits, because usually only a few hyperparameters matter much.

Python RandomizedSearchCV with distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use loguniform for hyperparameters that span orders of magnitude (C, alpha, learning rates) and randint or uniform for ranges. Always set random_state for reproducible searches.

Successive halving

HalvingGridSearchCV and HalvingRandomSearchCV start every candidate with a small budget (for example few samples), keep the best fraction, and give the survivors more data in each round. They can be several times faster on big problems. They currently live behind an experimental import:

Python
from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV

Good habits

Safe tuning workflow

Lock a test set Never touch it while tuning.
Search inside CV Use the training part only.
Score once Evaluate the winner on the test set.

The test score is only honest if it was not used to choose anything.

1 / 2
  • Search the whole pipeline, not just the last step, so scaling and imputation are refitted per fold.
  • Match the scorer to the goal. scoring="f1" or "roc_auc" for imbalanced problems; pass a list plus refit="f1" to track several metrics.
  • Fix random_states so results can be reproduced.
  • Search edges: if the best value sits at the edge of your range, extend the range and search again.
  • To get an unbiased estimate of the entire tuning procedure, wrap the search in cross_val_score (nested cross-validation).
Python Nested cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each outer fold runs a complete search on its own training part, then scores on its own held-out part, so the estimate includes the effect of tuning.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you refer to the C parameter of a step named svc inside a Pipeline grid?

    Pipeline parameters use the step name, a double underscore, then the parameter name.
  2. 2. When is RandomizedSearchCV usually a better choice than GridSearchCV?

    A grid grows multiplicatively. Random sampling covers continuous ranges with a fixed budget of n_iter fits.
  3. 3. Why is search.best_score_ an optimistic estimate of future performance?

    Selecting the best of many candidates by their CV score biases that score upward. Use a held-out test set or nested CV.
  4. 4. Which distribution suits a regularization strength C that could be anywhere from 0.001 to 100?

    A log-uniform distribution samples each order of magnitude evenly.

Summary

  • Hyperparameters are set before training; search them with cross-validation.
  • GridSearchCV tries every combination; RandomizedSearchCV samples n_iter settings from distributions.
  • Prefix pipeline parameters with the step name and __; search the whole pipeline.
  • Read cv_results_ and std_test_score; do not over-trust small differences.
  • Keep a test set outside the search, or use nested cross-validation for an honest estimate.

Next: Learning and Validation Curves