Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 40 ยท XGBoost API

Early Stopping, Cross-Validation and Saving Models

Pick the number of boosting rounds with xgb.cv and early stopping, then save and reload XGBoost models in JSON or UBJ and with the scikit-learn API, and avoid the pickle pitfalls.

You now know how to train XGBoost models two ways. Three practical skills remain before the library feels complete: choosing how many trees to build, estimating performance without wasting data, and saving a trained model so another process can use it. This lesson covers xgb.cv, the early-stopping rules, and the model file formats.

Choosing the number of rounds with xgb.cv

xgb.cv runs k-fold cross-validation inside XGBoost. At every boosting round it records the mean and standard deviation of your metric across folds, so you can see the whole learning curve and find where it bottoms out. With early_stopping_rounds it stops once the validation metric has not improved for that many rounds and returns the history up to the best round.

Python xgb.cv to find the number of rounds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recipe: run xgb.cv with a large num_boost_round, take the round count it settles on, and train the final model on all the training data with that many rounds. Because the final model sees more data than each fold did, some people add 5 to 10 percent more rounds; that is a judgment call, not a rule.

A word of caution about leakage: the number of rounds picked by CV is itself tuned on that data. For an honest estimate of future performance, keep a separate test set untouched until the very end (the habit from Splits, Baselines and Metrics).

Early stopping details

Whichever API you use, early stopping follows the same logic:

  1. After each round the validation metric is computed.
  2. If it is better than the best so far, the round is remembered.
  3. If early_stopping_rounds consecutive rounds fail to improve it, training stops.

Things that surprise people:

  • It uses the last evaluation set and the last metric. Reorder them on purpose.
  • best_iteration is zero-based. Using iteration_range=(0, best_iteration + 1) predicts with exactly the best trees.
  • There is no minimum-improvement threshold by default; a noisy metric can stop too early. Use a generous patience (20 to 50 rounds at eta=0.1) with small learning rates.
  • The validation set used for early stopping is not clean test data; reserve a separate one for the final score.
Python Early stopping and iteration_range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training stops 25 trees after the best one, so the booster holds 25 extra trees that did not help (in one run it stopped after 96 trees with the best at tree 71, and the all-trees logloss was only slightly worse). Recent versions use the best trees by default in predict, but being explicit with iteration_range is a good habit when you are unsure.

Saving and loading

XGBoost offers portable formats written by save_model. The extension picks the format:

File Format Notes
model.json JSON human readable, stable across versions, larger
model.ubj Universal Binary JSON same content, smaller and faster
model.model (legacy) old binary avoid for new work
Python Save, reload and verify predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the scikit-learn wrapper you normally call clf.save_model("clf.json") and later XGBClassifier().load_model("clf.json"), which also stores the class information. (The XGBoost 2.1 build in this site's browser runner cannot run save_model on the wrapper together with the newest scikit-learn, which is why the example above saves the wrapper's Booster instead; on a normal install both work.) save_model stores the trees and objective but not your preprocessing and not all wrapper attributes. If you used a Pipeline, save the whole pipeline (see below), or save the XGBoost part with save_model and your preprocessing separately and version them together.

Pickle and joblib

Pickling works, and it is the easy way to store a whole scikit-learn Pipeline. Its drawbacks are real, however:

  • It is tied to the library versions and Python version that wrote it; loading elsewhere can fail or change behavior.
  • Loading an untrusted pickle executes arbitrary code. Only unpickle files you created yourself.

A reasonable production pattern is: model in JSON/UBJ with save_model, preprocessing and feature list in their own versioned files, and the library versions pinned in a requirements file. The deployment chapter returns to this.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does xgb.cv return when early_stopping_rounds is used?

    You get a table of the learning curve. Its length is the number of rounds to use for the final model.
  2. 2. After xgb.cv suggests 140 rounds, how do you build the final model?

    CV is used to choose the number of rounds; the final model is trained once on all the training data.
  3. 3. Which is the safest way to store just an XGBoost model for use across versions?

    The JSON and UBJ formats are designed to be portable and do not execute code when loaded.
  4. 4. Why is unpickling a file from an unknown source dangerous?

    Pickle can call arbitrary callables while loading, so only load pickles you trust.

Summary

  • xgb.cv gives a per-round learning curve with fold means and standard deviations; use it to choose num_boost_round.
  • Early stopping watches the last evaluation set and last metric and stops after early_stopping_rounds rounds without improvement.
  • Use iteration_range to predict with exactly the best trees.
  • Save models with save_model in JSON or UBJ; the wrapper has the same methods.
  • Pickle is convenient for pipelines but version-fragile and unsafe for untrusted files.

Next: LightGBM Scikit-learn API