Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 40 ยท XGBoost API

XGBoost Scikit-learn API

Use XGBClassifier and XGBRegressor like any scikit-learn estimator: fit with an eval set, read learning curves, use pipelines and cross-validation, and clone, tune and inspect models.

XGBoost has two interfaces. The scikit-learn API (XGBClassifier, XGBRegressor, XGBRanker) behaves like every other estimator, so it slots into pipelines, cross-validation and grid search. The native API (DMatrix and xgb.train) exposes everything with a bit more ceremony; the next lesson covers it. Start with the scikit-learn API: it is what you will use most of the time.

Because the wrapper needs scikit-learn to be installed, the examples on this page import it first. In your own projects, install both packages (pip install xgboost scikit-learn).

The estimators

Class Task Default objective
XGBRegressor numeric target reg:squarederror
XGBClassifier binary or multiclass binary:logistic or multi:softprob (chosen from y)
XGBRanker learning to rank rank:ndcg (needs groups)
XGBRFClassifier / XGBRFRegressor random-forest style (many parallel trees, no boosting) like above

All of them take the usual parameters in the constructor, and every constructor argument is a plain attribute, so get_params(), set_params() and sklearn.base.clone work.

Python Fit, evaluate and read the learning curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several things are worth noticing:

  • eval_set takes a list of (X, y) pairs. They are named validation_0, validation_1, and so on, in that order.
  • Early stopping watches the last pair in eval_set and the last metric. Put the validation set last. In XGBoost 2.x early_stopping_rounds is a constructor argument (older code passed it to fit).
  • After early stopping, predict uses the best iteration automatically, and best_iteration and best_score are set.
  • verbose=False silences per-round logging; use verbose=10 to print every 10 rounds.

Never pick the number of trees on the test set. The validation set used for early stopping is part of training in the sense that it influences the model, so keep a separate final test set (see Splits, Baselines and Metrics).

Pipelines and cross-validation

Because the wrapper is a normal estimator, you can combine it with preprocessing. This pipeline imputes, scales a few columns and trains XGBoost, all evaluated with stratified cross-validation so that no preprocessing statistic leaks from validation folds.

Python XGBoost inside a scikit-learn Pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Trees do not need scaling, so the imputer is optional here (XGBoost handles missing values itself, see XGBoost Categorical and Missing Values). The pattern is what matters: put the preprocessing inside the pipeline so cross-validation refits it on each training fold.

Inspecting and cloning

Python Parameters, importances and clone

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

feature_importances_ is normalized to sum to 1 and defaults to the gain measure for tree boosters in recent versions; we compare importance types in the Feature Importance and SHAP chapter of the course.

Useful constructor arguments

Argument Purpose
n_jobs threads for training and prediction (-1 for all cores)
tree_method "hist" (the default in 2.x), "exact" for small data, "approx"
device "cpu" or "cuda" for GPU training
enable_categorical use pandas category columns directly
missing the value that means "missing" (default NaN)
random_state seeds sampling for reproducibility
sample_weight (in fit) per-row weights
flowchart LR
  A["Raw DataFrame"] --> B["ColumnTransformer<br/>impute and encode"]
  B --> C["XGBClassifier.fit"]
  C --> D["predict / predict_proba"]
  C --> E["evals_result, best_iteration"]

The wrapper turns the arrays into an internal DMatrix on every call, which is convenient but repeats work in tight loops.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In eval_set=[(X_tr, y_tr), (X_va, y_va)], which pair does early stopping watch?

    XGBoost uses the last dataset in the list (and the last metric), which is why the validation set should come last.
  2. 2. Where should preprocessing such as imputation go when you cross-validate?

    Fitting a preprocessor on all data leaks information from validation rows into training.
  3. 3. What does model.evals_result() return?

    It maps validation_0, validation_1, ... to dictionaries of metric lists with one value per round.
  4. 4. Which call gives you an unfitted copy of a model with the same parameters?

    clone builds a new estimator from get_params(), without any learned trees.

Summary

  • XGBClassifier, XGBRegressor and XGBRanker follow the scikit-learn estimator protocol.
  • Pass eval_set to fit and set early_stopping_rounds in the constructor; the last pair and last metric decide when to stop.
  • evals_result(), best_iteration and best_score describe the training run.
  • Put preprocessing inside a Pipeline for honest cross-validation.
  • get_params, set_params and clone make grid and random search straightforward.

Next: DMatrix and the Native train API