Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 40 ยท Hyperparameter Tuning

Learning Rate and the Number of Trees

Understand the trade-off between learning rate and number of boosting rounds, read learning curves, and use early stopping in XGBoost and LightGBM to pick the tree count.

In gradient boosting every new tree corrects the mistakes of the trees before it, and the learning rate (also called eta, learning_rate or shrinkage) controls how much of each correction is applied. A large rate makes big steps and fits fast, but it can overshoot and overfit. A small rate makes cautious steps, needs many more trees, and usually generalizes a little better.

The number of trees (n_estimators, num_boost_round, num_iterations) is therefore tied to the learning rate. This lesson shows how the two interact, how to read a learning curve, and how early stopping removes the guesswork.

How the two parameters interact

The prediction of a boosted model is a sum: the starting value plus learning_rate times the output of each tree. Halving the learning rate roughly doubles the number of trees needed to reach the same fit. Because of that, tuning is simplest in two steps:

  1. Pick a learning rate that your hardware can afford (0.05 to 0.1 for exploration, 0.01 to 0.03 for a final model).
  2. Let validation loss decide how many trees to build.
flowchart TD
  A["High learning rate<br/>big steps"] --> B["Few trees, fast fit"]
  A --> C["Risk: noisy, overshoots"]
  D["Low learning rate<br/>small steps"] --> E["Many trees, slow fit"]
  D --> F["Usually smoother and a bit more accurate"]

The two parameters trade off: lowering one means raising the other.

Early stopping

With early stopping, training watches a validation metric after every round and stops when it has not improved for early_stopping_rounds consecutive rounds. The best round is stored as best_iteration. In recent XGBoost versions you give early_stopping_rounds (and the eval_metric) to the constructor and pass eval_set to fit:

Python Early stopping at three learning rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the pattern clearly: the best iteration grows from about 120 at eta=0.5 to roughly 880 at eta=0.02, while the validation AUC stays within about 0.001 of each other. On this dataset the three rates end up equally good, and the small rate only costs more compute. That is common. A small learning rate pays off most on noisy data and large datasets, and when you want the last fraction of accuracy.

Set n_estimators far above what you expect to need (1000 to 5000) when you use early stopping. It is only a ceiling. Setting it too low silently cuts training short before the validation metric bottoms out.

Reading a learning curve

The evals_result() method returns the metric after every round for each set in eval_set. Plotting (or printing) train and validation loss together shows what is going on:

Python Train and validation loss per round

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training loss keeps falling forever (the trees can always memorize more), while the validation loss falls, flattens, and then slowly rises. Here it bottoms out near round 162 at about 0.302, and by round 400 it has climbed back to roughly 0.319. Training past the minimum is pure overfitting. Early stopping finds that minimum automatically.

The same idea in LightGBM

LightGBM uses callbacks for early stopping, and the scikit-learn wrapper accepts eval_set, eval_metric and callbacks in fit:

Python
import lightgbm as lgb

model = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.03, num_leaves=15)
model.fit(
    X_tr, y_tr,
    eval_set=[(X_va, y_va)],
    eval_metric="auc",
    callbacks=[lgb.early_stopping(stopping_rounds=50), lgb.log_evaluation(100)],
)
print(model.best_iteration_)

Names differ slightly between the libraries: XGBoost uses best_iteration and best_score, LightGBM uses best_iteration_ and best_score_ on the scikit-learn wrapper. With the native APIs (xgb.train, lgb.train) the number of rounds is passed as num_boost_round.

Details that bite

  • Which metric stops training? When you list several metrics, early stopping uses the last one in XGBoost and by default all of them in LightGBM (first_metric_only=True restricts it). Make the metric you care about explicit.
  • Patience. With a small learning rate, validation noise lasts longer, so use more patience (50 to 200 rounds). With eta=0.5 ten rounds can be enough.
  • Which set stops training? If you pass several sets in eval_set, XGBoost stops on the last one. Put the real validation set last, after any training-set entry used for plotting.
  • Refitting. After early stopping the model already keeps all rounds but predicts with the best one in recent versions. To train on train plus validation rows, refit with n_estimators set to best_iteration (a little more, such as 10% extra, to account for the bigger dataset).
  • Cross-validation. xgb.cv and lgb.cv run early stopping inside the folds and give a mean best round to use for the final model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You halve the learning rate and keep everything else the same. What usually has to change?

    Each tree contributes learning_rate times its output, so smaller steps need more trees to cover the same distance.
  2. 2. What does early_stopping_rounds=50 mean?

    Training continues until the validation score fails to improve for that many rounds in a row; the best round is remembered.
  3. 3. In a learning curve, the training loss keeps falling while the validation loss starts rising. This indicates:

    The model is memorizing the training data. The round where validation loss is lowest is the right stopping point.
  4. 4. Why should n_estimators be set high when you use early stopping?

    Early stopping decides the real number of trees. A low ceiling would cut training short.

Summary

  • Learning rate scales each tree's contribution; smaller rates need more trees and are a little more robust.
  • Use early stopping on a held-out validation set instead of guessing n_estimators.
  • Learning curves show training loss falling forever and validation loss bottoming out at the best round.
  • XGBoost passes early_stopping_rounds to the constructor; LightGBM uses lgb.early_stopping as a callback.
  • Choose a learning rate you can afford, tune other settings with it fixed, then lower it for the final model if time allows.

Next: Tree Complexity Parameters