Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 40 ยท Hyperparameter Tuning

Tuning LightGBM and XGBoost Side by Side

A parameter translation table between XGBoost and LightGBM, native cv with early stopping in both libraries, and starting ranges that give a strong first model.

XGBoost and LightGBM solve the same problem with the same core algorithm, but their parameter names, defaults and native APIs differ. Moving a tuned configuration from one library to the other without translating it is a common source of confusing results. This lesson gives you a translation table, shows the native cv functions of both libraries, and ends with starting ranges you can use on a new dataset.

Parameter translation

Meaning XGBoost (sklearn name) LightGBM (sklearn name) Note
Step size learning_rate / eta learning_rate XGBoost default 0.3, LightGBM default 0.1
Number of trees n_estimators n_estimators native: num_boost_round
Tree size max_depth (default 6) num_leaves (default 31) different growth strategies
Row sampling subsample subsample / bagging_fraction LightGBM also needs subsample_freq above 0
Column sampling colsample_bytree colsample_bytree / feature_fraction
Min evidence per leaf min_child_weight min_child_samples hessian sum versus row count
L2 penalty reg_lambda (default 1) reg_lambda (default 0) defaults differ
L1 penalty reg_alpha (default 0) reg_alpha (default 0)
Min split gain gamma min_split_gain
Histogram bins max_bin (hist method) max_bin (default 255)
Class weight scale_pos_weight scale_pos_weight, is_unbalance
Thread count n_jobs n_jobs

The most important traps are on this list. LightGBM ignores subsample unless subsample_freq is at least 1. XGBoost uses L2 regularization by default but LightGBM does not. And a max_depth of 6 in XGBoost means up to 64 leaves, while LightGBM's default 31 leaves can be any depth.

The same data, matched settings

The next example trains both libraries with roughly equivalent settings: depth 5 (up to 32 leaves) versus num_leaves=31, the same learning rate, row and column sampling, and the same L2 penalty.

Python Matched XGBoost and LightGBM models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the AUCs were 0.959 (XGBoost) and 0.962 (LightGBM), with fit times of a few seconds each. The two scores are normally very close. Differences of a few thousandths between libraries are within the noise of the split, so choose by speed, features (categoricals, GPU, ranking) and what your team already deploys, not by one benchmark.

Native cross-validation with early stopping

Both libraries have a native cv function that builds the folds, runs early stopping on every fold, and returns the metric per round. This is the quickest way to find the number of trees for fixed parameters.

Python xgb.cv and lgb.cv find the tree count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice small API differences. xgb.cv returns a DataFrame and takes early_stopping_rounds directly; lgb.cv returns a dictionary and uses a callback. In some LightGBM versions the key is valid auc-mean and in others auc-mean, so the code above finds it by suffix. The length of the result is the best number of rounds, because both stop at the best iteration (after trimming the extra patience rounds).

When you move from cv to a final model, set n_estimators to the best round count, times about 1.1 if you train on more rows than a single fold used.

Starting ranges

A good first search space for most tabular problems, with a learning rate of 0.05 and early stopping:

flowchart TD
  A["Fix learning_rate 0.05<br/>n_estimators 3000 + early stopping"] --> B["Tree size:<br/>max_depth 3-8 or num_leaves 8-64"]
  B --> C["Evidence:<br/>min_child_weight 1-20<br/>min_child_samples 10-100"]
  C --> D["Randomness:<br/>subsample 0.6-1.0<br/>colsample_bytree 0.4-1.0"]
  D --> E["Penalties:<br/>reg_lambda 0.1-30<br/>reg_alpha 0-5"]
  E --> F["Lower the rate to 0.02<br/>refit with early stopping"]

A typical search order, from the largest effects to the smallest.

Do not search every parameter at once on small data. Four or five well-chosen parameters with 30 to 60 trials beat a ten-dimensional space with the same budget.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You set subsample=0.7 in LightGBM and see no change in the results. What is the most likely reason?

    LightGBM only bags rows when subsample_freq (bagging_freq) is at least 1.
  2. 2. Which pair of parameters plays the same role of limiting tree size, in XGBoost and LightGBM respectively?

    XGBoost grows level by level and is limited by max_depth; LightGBM grows leaf by leaf and is limited by num_leaves.
  3. 3. What is a difference between the default regularization of the two libraries?

    XGBoost applies an L2 penalty of 1 by default; LightGBM defaults to no penalty.
  4. 4. What does the length of the result of xgb.cv with early stopping tell you?

    Early stopping truncates the history at the best round, so its length is the recommended number of rounds.

Summary

  • The libraries share ideas but not names or defaults: translate parameters before comparing models.
  • XGBoost is sized by max_depth, LightGBM by num_leaves; evidence per leaf is min_child_weight versus min_child_samples.
  • LightGBM needs subsample_freq for row sampling, and its default L2 is 0.
  • xgb.cv and lgb.cv give fold-based early stopping and a recommended tree count.
  • Start with learning rate 0.05, tune tree size and evidence first, penalties last.

Next: Feature Importance: Gain, Weight and Cover