Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 40 ยท LightGBM and CatBoost

LightGBM Scikit-learn API

Train LightGBM with LGBMClassifier and LGBMRegressor, use callbacks for early stopping and logging, read evals_result_, and set the parameters that matter most for speed and overfitting.

LightGBM (Light Gradient Boosting Machine) comes from Microsoft and shares XGBoost's second-order gradient boosting core. What sets it apart is speed on large data and its leaf-wise tree growth, which you met in Histogram Splits and Tree Growth Strategies. Like XGBoost it has a scikit-learn style API (LGBMClassifier, LGBMRegressor, LGBMRanker) and a native one (lgb.Dataset, lgb.train). This lesson covers the first, and the next one the second.

A first model

Python LGBMClassifier with an evaluation set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Differences from XGBoost worth memorizing:

  • Callbacks, not constructor arguments. Early stopping and logging are callbacks passed to fit: lgb.early_stopping(...), lgb.log_evaluation(period) and lgb.record_evaluation(dict).
  • Trailing underscores. Fitted attributes end with _, such as best_iteration_, best_score_, evals_result_ and feature_importances_ (scikit-learn convention).
  • Early stopping checks every metric by default (first_metric_only=False), unlike XGBoost's last-metric rule. Set first_metric_only=True in the callback to watch only the first.
  • Quiet output. verbose=-1 removes the "No further splits" and info messages that otherwise fill the console.

The parameters that matter most

Parameter Default What it does
n_estimators 100 number of boosting rounds
learning_rate 0.1 shrinkage of every tree
num_leaves 31 maximum leaves per tree (the main complexity control)
max_depth -1 depth cap (-1 means no limit)
min_child_samples 20 minimum rows in a leaf
subsample / subsample_freq 1.0 / 0 row sampling (needs subsample_freq >= 1)
colsample_bytree 1.0 fraction of features per tree
reg_alpha, reg_lambda 0, 0 L1 and L2 penalties
min_split_gain 0 minimum gain to split (like XGBoost's gamma)
class_weight None "balanced" for imbalanced classes
n_jobs -1 threads

The aliases can be confusing: the same parameter may be called bagging_fraction, subsample or sub_row. The scikit-learn names above are the safest to use; mixing two aliases of one parameter makes LightGBM keep one and warn.

Regression and the learning curve

LGBMRegressor defaults to the L2 (squared error) objective. Use evals_result_ to plot or print the validation curve.

Python Regression with a recorded learning curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you give two evaluation sets, early stopping looks at all of them in LightGBM unless told otherwise, but it ignores the training set (it never stops on training data). Naming sets with eval_names makes the printed history readable.

Predicting and reading importances

predict_proba, predict and the num_iteration argument work as you would expect. feature_importances_ uses split counts by default (importance_type="split"); change it to "gain" when you build the model for total improvement.

Python Importance by split count vs gain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two rankings often differ: a feature used in many small splits ranks high by count, while one used rarely but decisively ranks high by gain. Neither measures causal importance; Chapter 6 shows better tools.

flowchart LR
  D["DataFrame / array"] --> F["LGBMClassifier.fit<br/>eval_set + callbacks"]
  F --> B["booster: best_iteration_"]
  F --> H["evals_result_"]
  B --> P["predict / predict_proba"]

Callbacks plug into fit; the fitted object keeps the best iteration and history.

Quick pitfalls

  • If you see [LightGBM] [Warning] No further splits with positive gain, your trees ran out of useful splits (a sign that you can stop earlier or lower n_estimators).
  • Column names containing special JSON characters (such as ", ,, :, [) raise errors; clean them first.
  • Passing a DataFrame keeps the column names for importances; passing a NumPy array names them Column_0, Column_1, and so on.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you enable early stopping with LGBMClassifier.fit?

    Modern LightGBM uses callbacks for early stopping, logging and recording the history.
  2. 2. Which LightGBM parameter is the main control of tree size?

    Trees grow leaf-wise, so the number of leaves is the key complexity knob.
  3. 3. Why might subsample=0.8 not change a LightGBM model?

    Row subsampling starts only when subsample_freq is at least 1.
  4. 4. What does best_iteration_ hold after early stopping?

    predict uses that many trees by default, and the attribute is available after fit.

Summary

  • LGBMClassifier, LGBMRegressor and LGBMRanker follow the scikit-learn protocol; fitted attributes end in _.
  • Early stopping, logging and history recording are callbacks passed to fit.
  • num_leaves is the main size control; remember subsample_freq for bagging.
  • Use evals_result_ and named evaluation sets to read learning curves.
  • Importance by split and by gain can disagree, so compare both.

Next: LightGBM Dataset and Categorical Features