Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Feature Importance and SHAP

Dependence, Interactions and Monotone Constraints

See how a boosted model reacts to one feature with partial dependence, find interactions with pred_interactions, and force sensible monotone effects in XGBoost and LightGBM.

Importance tells you which features matter. This lesson shows how they matter: the shape of the effect of a feature, whether two features interact, and how to tell the model that an effect must go in one direction. These tools turn a black box into something you can sanity-check against domain knowledge.

Partial dependence

A partial dependence curve answers: "if I set this feature to a value v for every row (leaving the other columns as they are) and average the predictions, how does the average change with v?" It shows the average effect of one feature. Scikit-learn computes it for any model with sklearn.inspection.partial_dependence:

Python Partial dependence of a non-linear effect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see x1 rise and fall like a sine wave, x2 increase almost linearly, and x3 stay almost flat. The shape of the true relationship is recovered without you ever specifying its form: that is the strength of trees. Two cautions: the curve is an average, so it can hide opposite effects in different subgroups (ICE curves, which draw one line per row, reveal that), and it evaluates impossible combinations if features are strongly correlated.

Interactions

Boosted trees with depth above 1 can learn interactions: the effect of one feature depends on another. XGBoost can break SHAP values down into main effects (diagonal) and pairwise interaction effects (off-diagonal) with pred_interactions=True. The result has shape (rows, features + 1, features + 1).

Python Finding an interaction with pred_interactions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The off-diagonal entry for the pair (a, b) is large, while pairs with c are close to zero: the model discovered that a and b work together, and that c acts alone. The check at the end confirms the decomposition: summing the interaction row of a feature gives its ordinary SHAP value.

Interaction values cost much more to compute than ordinary SHAP values (quadratic in the number of features). Compute them on a sample of a few hundred or thousand rows, and only for a model with a modest feature count.

Monotone constraints

Sometimes you know a relationship must be monotone: more income should never decrease the credit score, a higher price should never increase predicted demand. Unconstrained trees may still produce small violations because of noise, and those violations look wrong to users and regulators. Both libraries let you impose the rule at training time:

  • XGBoost: monotone_constraints=(1, -1, 0) (one entry per feature in column order; 1 increasing, -1 decreasing, 0 free).
  • LightGBM: monotone_constraints=[1, -1, 0] with monotone_constraints_method set to "basic", "intermediate" or "advanced" (stricter methods are less restrictive but slower).
Python A noisy feature, with and without a monotone constraint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unconstrained model wiggles and has about twenty downward steps (out of 39) along a feature that truly only increases. The constrained model has none. Constraints usually cost a very small amount of accuracy, sometimes none at all, and they can help generalization because they encode correct prior knowledge. Use them for features where the direction is a business rule, not a guess.

Interaction constraints

You can also restrict which features may appear together in one tree path with interaction_constraints (XGBoost, for example [[0, 1], [2, 3]]; LightGBM accepts the same idea). Then features in different groups can never interact. This keeps the model additive between groups, which improves interpretability and sometimes robustness.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a partial dependence curve show?

    Partial dependence sets the feature to each grid value for all rows and averages the predictions.
  2. 2. What does pred_interactions=True in XGBoost return?

    The array has shape (rows, features+1, features+1); off-diagonal entries are pairwise interaction effects.
  3. 3. What does monotone_constraints=(1, -1, 0) mean for three features?

    1 forces a non-decreasing effect, -1 a non-increasing effect, and 0 leaves the feature free.
  4. 4. Why can partial dependence be misleading?

    An average can mask heterogeneity; ICE curves show individual rows.

Summary

  • Partial dependence shows the average effect shape of one feature; ICE curves show individual rows.
  • pred_interactions=True splits SHAP values into main and pairwise interaction effects.
  • Monotone constraints force a direction on a feature's effect; they are cheap and often harmless.
  • Interaction constraints limit which features can be combined in a tree path.

Next: Class Imbalance and Sample Weights