Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 40 ยท Feature Importance and SHAP

Feature Importance: Gain, Weight and Cover

Learn the built-in importance types of XGBoost and LightGBM (weight, gain, cover, split), why they disagree, and how to plot and interpret them without being misled.

After a model is trained, the first question is usually "which features matter?". Boosted trees answer it cheaply: every split in every tree uses a feature, so you can count or sum something per feature. But there are several ways to count, and they can rank the same features very differently. This lesson explains each built-in measure and its biases, so you know what you are looking at.

Three ways to count

For every split in the ensemble, XGBoost records which feature was used, the loss reduction (gain) the split achieved, and the number of training rows (cover) that passed through it. The built-in importance types are summaries of those records:

Type XGBoost name LightGBM name Meaning
How often weight split number of times the feature is used in a split
Average benefit gain not default mean loss reduction per split that uses the feature
Total benefit total_gain gain sum of loss reductions over all splits using the feature
Rows affected cover not default mean number of rows (or hessian sum) at the feature's splits

Total gain is usually the most meaningful of the four. It answers "how much did this feature reduce the training loss overall?". Weight is the weakest: a feature used in many trivial splits looks important even if each split helps little.

Computing them

Python Compare importance types on the same model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several things to read from the output. The four informative features (f0 to f3) dominate total gain. The noise features still get nonzero weight, because trees will happily split on noise late in training, but their gain is small. Finally, feature_importances_ on the scikit-learn wrapper is a normalized version of one type; the default in recent XGBoost versions is gain, while LightGBM's default is split. If you compare the two libraries, set importance_type explicitly.

Same idea in LightGBM

Python
import lightgbm as lgb
model = lgb.LGBMClassifier(n_estimators=150, importance_type="gain", verbose=-1).fit(X, y)
print(dict(zip(X.columns, model.feature_importances_.round(1))))
# or, from the booster:
model.booster_.feature_importance(importance_type="split")

The LightGBM plotting helper lgb.plot_importance(model, importance_type="gain") and XGBoost's xgb.plot_importance(model, importance_type="gain") draw a bar chart with matplotlib.

Biases to remember

Built-in importances are computed on the training data and from the structure of the trees, which leads to known problems:

  • High-cardinality bias. A continuous or ID-like feature offers many candidate thresholds, so it has more chances to find a lucky split. Noise features with many distinct values often outrank binary features that are truly informative.
  • Correlated features share credit unpredictably. If two features carry the same signal, the trees pick one at random per split, and the credit is divided arbitrarily. Removing one often makes the other jump up.
  • No direction, no size. An importance of 0.3 says the feature is used a lot, not whether higher values raise or lower the prediction.
  • Training-set bias. A feature that only helps to memorize the training set still shows high gain.

The next experiment demonstrates the first bias: a noise column with many unique values against a genuinely useful binary feature.

Python A noisy continuous column can outrank a useful binary one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The noise column collects about nine times the weight of the useful feature (roughly 2,500 splits against 285) and even more total gain, although it carries no signal. On validation data it does nothing; the AUC comes entirely from the binary feature. This is why the next lessons introduce permutation importance, which is measured on held-out data, and SHAP values, which are exact attributions per prediction.

Practical advice

  • Prefer total_gain (or gain in LightGBM) over weight for a quick look.
  • Add a random noise column during experiments. Any real feature that ranks below the noise column is probably useless.
  • Report importance with caution in business settings: "feature X is used a lot by the model" is not "feature X causes the outcome".
  • Use importances for debugging and feature selection (an ID column that ranks first is a leak, see pitfalls), not as final explanations.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which built-in importance type counts how many times a feature is used in a split?

    Weight in XGBoost and split in LightGBM are plain counts of splits using the feature, regardless of how useful those splits were.
  2. 2. Why can a pure noise feature with many unique values rank high in built-in importance?

    More distinct values mean more split candidates, so a noise feature will find splits that reduce training loss by chance.
  3. 3. Two features carry almost the same information. What usually happens to their importances?

    Trees pick between nearly equivalent features at random, so credit is divided unpredictably and can shift with the seed.
  4. 4. Which statement about built-in importances is true?

    Built-in importances only describe how much the trees use a feature, not the direction of the effect or causality.

Summary

  • Importance types are summaries of tree splits: weight (count), gain (loss reduction), cover (rows).
  • total_gain is the most informative built-in measure; weight is easiest to fool.
  • Defaults differ: feature_importances_ is gain-based in XGBoost and split-based in LightGBM.
  • Biases include favoring high-cardinality features, arbitrary credit sharing among correlated features, and no direction.
  • Treat them as a debugging aid; use permutation importance and SHAP for stronger claims.

Next: Permutation Importance