Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 40 ยท Hyperparameter Tuning

A Sensible Tuning Workflow

Learn how to tune gradient boosting models without fooling yourself: baselines, train/validation/test splits, early stopping, search spaces and how much tuning is worth.

Gradient boosting libraries expose dozens of parameters, and it is tempting to throw a giant grid at them and wait. That is rarely the best use of your time. A good tuning process starts with a strong baseline, protects you from overfitting to your own validation data, and spends the search budget where it matters.

This lesson sets up the workflow that the rest of the chapter builds on. Every later lesson (learning rate, tree size, regularization, search methods) plugs into the loop you see here.

Start from a baseline

Before changing anything, train the model with default parameters and record a validation score. The baseline answers two questions: "how good is good enough?" and "did my tuning actually help?". Both XGBoost and LightGBM have defaults that were chosen to work reasonably on many datasets, so the baseline is often already close to the best you will get.

You also need three disjoint sets of rows:

  • Train: the data the trees are fitted on.
  • Validation: the data you look at over and over while choosing parameters and the number of trees.
  • Test: the data you touch once, at the very end, to report an honest number.

If you tune on the test set, its score becomes an optimistic estimate, because you kept the configuration that happened to fit its noise. With small datasets use cross-validation instead of one validation split, but keep a final test set aside.

Python Three splits and a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the gap: the training AUC is 1.0 while the validation AUC is about 0.96. The model memorizes the training rows. That gap is the signal that regularization and early stopping have something to work with, but it does not by itself mean the validation score is bad.

The tuning loop

The idea is always the same: propose parameters, fit on train, score on validation, keep the best, and only then look at the test set once.

flowchart LR
  A["Baseline with defaults"] --> B["Pick a search space"]
  B --> C["Fit on train, early-stop on validation"]
  C --> D["Record best score and best rounds"]
  D -->|"more budget"| B
  D -->|"done"| E["Refit with best settings"]
  E --> F["Score the test set once"]

The test set appears only at the very end of the loop.

Two details make this loop efficient with boosted trees:

  1. Fix a small learning rate range and let early stopping choose the number of trees. n_estimators is not really a hyperparameter to search over; set it high (for example 1000) and stop when the validation metric has not improved for 30 or 50 rounds. The stopping point (best_iteration) is then the number of trees to use when you refit. The next lesson goes deeper.
  2. Search the other parameters on a log scale where it makes sense. Learning rate and many regularization strengths act multiplicatively, so draw them as 10 ** uniform(-2, -0.5) rather than uniformly.

What to tune, in what order

Not all parameters matter equally. A practical order that works for both libraries:

  1. Learning rate and the number of trees (via early stopping).
  2. Tree complexity: max_depth / min_child_weight in XGBoost, num_leaves / min_child_samples in LightGBM (tree complexity lesson).
  3. Randomness: row subsampling and column subsampling (regularization lesson).
  4. L1/L2 penalties and the minimum split gain.
  5. Task-specific knobs: class weights, objective, evaluation metric (imbalanced data).

Random search draws each parameter independently from a distribution. It is simple, parallelizable, and beats a coarse grid when only a few parameters matter (a classic result: the grid wastes trials repeating the same values of the important parameters). The code below writes the search by hand, so nothing is hidden.

Python Random search with early stopping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what happens: the best of twelve random configurations scores about the same as the default model on validation (roughly 0.96). That is not a failure of the search. It is the honest outcome on a dataset where the defaults were already sound and where the validation set (800 rows) has a noise level of a few thousandths of AUC. Differences smaller than that noise are not real improvements.

How much is tuning worth?

Expect these rules of thumb:

  • Large wins come from fixing the problem setup: the right objective, no leakage, good features, correct handling of imbalance, early stopping.
  • Medium wins (1-3% of the metric) come from tuning tree size, learning rate and subsampling when the defaults overfit or underfit badly.
  • Small wins (below the noise of your validation set) come from fine-grained searches over many parameters. If a search finds a "better" score only in the third decimal, rerun with a different seed before believing it.

Use cross-validation, or several random seeds, when you compare two settings whose scores differ by less than about 0.005. Report the mean and the spread, not a single run.

Common mistakes

  • Early stopping on the test set. If the test rows decide when training stops, they are no longer unseen.
  • Preprocessing on all rows before splitting. Fit encoders and imputers on train only (see pitfalls).
  • Comparing runs with different splits. Keep the split fixed (or use the same CV folds) while comparing parameter sets.
  • Treating n_estimators as a free parameter next to a fixed learning rate: those two trade off against each other.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we keep a test set that is never used while tuning?

    Every time you compare configurations on a set, you select for configurations that fit its noise. A set that is used only once at the end gives an unbiased estimate.
  2. 2. Which parameter is best chosen by early stopping rather than by a grid?

    With a fixed learning rate, the right number of trees is the round where the validation metric stops improving, so you let early stopping find it.
  3. 3. Why is random search usually preferred over a coarse grid search?

    A grid repeats the same few values of each parameter. When only a few parameters matter, random sampling explores those dimensions much better for the same number of fits.
  4. 4. The best tuned model beats the baseline by 0.001 AUC on an 800-row validation set. What should you conclude?

    Validation scores on small sets fluctuate by a few thousandths. A difference that small needs repeated evaluation before you trust it.

Summary

  • Fit a default baseline first and keep train, validation and test sets separate.
  • Let early stopping pick the number of trees; search the other parameters, using log scales for rates and strengths.
  • Tune in order of impact: learning rate and trees, tree size, subsampling, penalties, task-specific settings.
  • Random search is a strong default; gains below the noise of your validation set are not real.
  • Touch the test set once, after the final refit.

Next: Learning Rate and the Number of Trees