Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 111 ยท Trees and Ensembles

Feature Importance and Interpretation

Find out which features a model uses: impurity importance and its pitfalls, permutation importance, partial dependence and ICE plots, and why correlated features fool them.

A model that predicts well is useful, but a model you can question is far more useful. Which columns drive its predictions? Does a higher value of a feature push the prediction up or down, and by how much? Answers to these questions help you debug a model, catch data leaks, explain results to colleagues, and decide which columns are worth collecting.

Tree ensembles hand you one answer for free (feature_importances_), and it is the one you should trust least. This lesson explains why, then teaches two better tools in sklearn.inspection: permutation importance, which says how much a feature matters to the score, and partial dependence (with ICE curves), which shows how the prediction changes with a feature. It finishes with the caveat that applies to all of them: correlated features.

Three questions, three tools

Impurity How much did splits on this feature clean up the data?
Permutation How much worse is the score if I shuffle it?
Partial dependence What does the prediction do as it changes?

Importance says how much. Partial dependence says which way.

1 / 2

Impurity importance and its pitfalls

Every tree ensemble in scikit-learn has a feature_importances_ attribute. It adds up, over all trees, how much each feature's splits reduced the impurity (Gini or squared error) and normalizes the totals to sum to 1. It costs nothing, and for a quick first look it is fine. But it has two well-known problems:

  1. It is computed on the training data. A feature the model uses to memorize noise still earns importance, even though it does nothing for new rows.
  2. It favors features with many distinct values. A column with a unique value per row (like an id) or a continuous column offers many places to split, so the trees find plenty of chance splits on it.

The experiment below shows both. One column carries real signal. Three are noise: a coin flip, a continuous random number and a shuffled row id. We train a forest and compare impurity importance with permutation importance computed on held-out data (explained in the next section).

Python Impurity importance gets fooled by noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forest memorizes its training rows (training accuracy far above test accuracy), and the impurity column hands a substantial share of importance to the noise columns, especially the continuous ones. The permutation column, measured on rows the forest never saw, puts the signal far ahead and the noise near zero. That is the gap you should expect whenever a model overfits.

Permutation importance

Permutation importance asks a direct question: how much does the model's score drop if this column stops carrying information? The recipe:

  1. Score the fitted model on a data set (ideally held-out data).
  2. Shuffle one column, which breaks its link with the target but keeps its distribution.
  3. Score again. The drop is that feature's importance.
  4. Repeat with several shuffles and average, to smooth out luck.

It works with any fitted model, not only trees, and it measures what you actually care about (the score). In scikit-learn it is permutation_importance(model, X, y, n_repeats=..., scoring=...), which returns importances_mean, importances_std and the raw importances array.

Python Permutation importance with spread

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three of the eight columns were built to be informative, but make_regression gives them random coefficients, and in this run only two (x4 and x7) carry real weight; the third is so weak it barely registers. Everything else sits at or very near zero. A negative value just means the shuffled column happened to help slightly by chance, so values close to zero mean "no evidence this feature matters".

A few practical rules:

  • Use held-out data (a validation or test set). On training data an overfit model reports importance for features it only memorized.
  • Use the metric you care about through scoring. Importance for accuracy and importance for recall can differ.
  • Keep n_repeats at 10 or more and look at the spread, not only the mean.
  • It costs one prediction pass per feature per repeat, so on wide data you may need a subsample.

Permutation importance describes this fitted model on this data. It does not prove that a feature causes the target. A feature can be important to a model simply because it is a good proxy for something else.

The correlated-features caveat

If two features carry the same information, shuffling one barely hurts the score, because the model can fall back on its twin. Permutation importance then splits the credit between them or hides both. Impurity importance does the same during training: the trees pick one twin or the other more or less at random.

In the next example the target depends on a (strongly) and b. We then add a_copy, a near duplicate of a, and watch what happens to the importance of a.

Python A near-copy dilutes importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model's accuracy is unchanged (R2 0.96 both times), yet the importance of a drops from about 1.7 to about 1.0, and its copy picks up only about 0.15. Together they look less important than a did alone. If you read the second table without knowing about the duplicate, you might wrongly conclude that a matters less.

What to do about it:

  • Group correlated features and shuffle them together, or drop all but one from each group and compare the scores.
  • Inspect a correlation matrix (or cluster the features) before interpreting any importance ranking.
  • Treat rankings among correlated features as unreliable, and say so when you report them.

Partial dependence and ICE

Importance tells you whether a feature matters. Partial dependence (PD) shows how: it varies one feature across its range, forces that value onto every row, and averages the model's predictions. The resulting curve answers, "on average, what does the model predict if this feature equals x?"

An ICE curve (individual conditional expectation) does the same for a single row instead of the average, so you get one line per row. Plotting ICE lines behind the PD curve shows whether the average hides different behavior in different rows. If the lines are parallel, the feature acts the same way everywhere. If they fan out or cross, the feature interacts with something else, and the average is misleading.

PartialDependenceDisplay.from_estimator draws both. Use kind="average" (the default), "individual" or "both".

Python Partial dependence with ICE lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data was built with a sine wave in x0 and a straight line in x1. The red PD curve for x0 recovers the wave, while the one for x1 is close to a straight rising line. In the x0 panel the thin ICE lines do not all follow the red curve: they fall into two groups with a small step on opposite sides of x0 = 0, because x0 interacts with x2 (the sign of their product adds a bump). A plain PD curve would have averaged that story away. With a limited subsample the plot stays readable and fast.

Partial dependence has its own caveat, the same one as before: it forces feature values onto every row independently of the others, so it can evaluate the model at combinations that never occur in real data (for example, a very tall child). When features are strongly correlated, read PD curves with caution.

A short interpretation workflow

  1. Fit and validate the model first. Interpreting a bad model explains nothing.
  2. Compute permutation importance on held-out data, with a metric that matches the goal.
  3. Check correlations among the top features, and group or drop duplicates.
  4. Draw partial dependence (with ICE) for the top few features to see the direction and shape.
  5. Sanity-check with domain knowledge. A top feature that should not matter is often a leak; see Nested Cross-Validation and Data Leakage later in this course.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can impurity-based feature_importances_ be misleading?

    Impurity importance reflects how the trees used each feature while fitting the training data, so a model that overfits can give noise features, especially continuous ones, a sizeable share.
  2. 2. What does permutation importance measure?

    Shuffling a column breaks its relationship with the target; the resulting drop in score is how much the model relied on it.
  3. 3. On which data should permutation importance normally be computed?

    On training data an overfit model shows importance for features it merely memorized. Held-out data shows what generalizes.
  4. 4. Two features are near-duplicates. What happens to their permutation importances?

    Shuffling one leaves the other available to the model, so the score hardly drops and neither feature looks as important as the shared information really is.
  5. 5. ICE lines that fan out or cross behind a partial dependence curve suggest what?

    Parallel ICE lines mean the feature acts the same way everywhere. Diverging lines show that its effect depends on other features.

Summary

  • feature_importances_ on trees is quick but computed on training data and biased toward high-cardinality or continuous columns.
  • permutation_importance shuffles a column and measures the score drop; use held-out data, a suitable scoring, and several repeats.
  • Correlated features share or hide importance; group them or remove duplicates before ranking.
  • PartialDependenceDisplay.from_estimator shows how predictions change with a feature; kind="both" adds ICE lines that reveal interactions.
  • Importance and dependence describe the model, not causation, and a surprising top feature is a signal to look for leakage.

Next chapter: separate the classes with the widest possible margin in Support Vector Machines.