Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Saving and Using Models

Inspecting and Explaining Models

Understand what a scikit-learn model learned: linear coefficients, tree feature importances, permutation importance on held-out data, partial dependence and ICE curves.

A model that predicts well but cannot be explained is hard to trust, debug or defend. Inspection answers questions like: Which features matter? In which direction? Does the model use something it should not, such as an ID column? Is the relationship what domain experts expect?

scikit-learn offers four layers of tools: coefficients for linear models, impurity importances for trees, permutation importance that works for any model, and partial dependence plots that show how predictions change with a feature. In this lesson you will use all four, and see where each can mislead.

Linear model coefficients

For a linear model, each coefficient is the change in the prediction for a one-unit increase in that feature, holding others fixed. To compare features, standardize them so a unit means "one standard deviation".

Python Reading standardized coefficients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The largest absolute values (here s5, s1 and bmi) matter most; the sign shows the direction. Notice that s1 is strongly negative while its relative s2 is positive: the blood-serum columns are correlated, so their coefficients partly cancel each other. Do not over-interpret one coefficient in isolation.

Tree importances

Random forests and gradient boosting expose feature_importances_: how much each feature reduced impurity across all splits. It is free and fast, but biased toward continuous and high-cardinality features and is computed on training data.

Python Impurity importance can be fooled

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The useless continuous column still gets a visible share, because there are many thresholds to find chance patterns. That is why you should also check permutation importance.

Permutation importance

Idea: shuffle one column in the test data and see how much the score drops. If shuffling barely matters, the model does not rely on it. It works for any model, uses your chosen metric, and measures reliance on held-out data.

Permutation importance

Score Measure on held-out data.
Shuffle Scramble one feature.
Drop Bigger drop means more important.

Repeat several times for a mean and a spread.

1 / 2
Python permutation_importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only signal produces a large drop; the two noise features sit close to zero, an order of magnitude below it (tiny values of either sign are just chance). Note: if two features are strongly correlated, shuffling one leaves the other to cover for it, so both can look unimportant. Group them or drop one and compare.

Partial dependence and ICE

Partial dependence (PDP) shows the average prediction as one feature varies while the others keep their actual values. ICE (individual conditional expectation) draws one line per row, showing whether the effect is the same for everyone.

Python Partial dependence values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Predictions rise with BMI in a roughly increasing but not perfectly straight way, something a coefficient alone cannot show. In notebooks, PartialDependenceDisplay.from_estimator(model, X, ["bmi", "s5"], kind="both") draws the plots (PDP plus ICE lines).

Python Plot PDP and ICE curves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the ICE lines fan out in different directions, the feature interacts with others and the average line hides that.

Per-prediction explanations and a word of caution

scikit-learn itself explains models globally. For explaining a single prediction, libraries such as SHAP or LIME are popular. For linear models you can compute it by hand: each feature's contribution is coef * (value - mean).

Keep in mind:

  • Importance shows what the model uses, not what causes the outcome.
  • Always compute permutation importance and PDP on held-out data.
  • A surprisingly important feature is a red flag for leakage (an ID, a timestamp, or a column derived from the target).
  • Combine inspection with error analysis: look at the rows with the biggest errors and ask what they have in common.
Python Error analysis: where does the model fail?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why standardize features before comparing linear coefficients?

    Coefficients depend on feature units. After standardizing they are comparable in size.
  2. 2. What is a known weakness of impurity-based feature_importances_?

    Features with many possible split points get inflated importance, even if useless.
  3. 3. How does permutation importance measure a feature?

    A feature the model relies on causes a large drop when its values are scrambled.
  4. 4. ICE lines in a partial dependence plot fan out in different directions. What does this suggest?

    PDP shows an average. ICE reveals heterogeneous effects hidden by that average.

Summary

  • Standardized coefficients explain linear models; correlated features make them unstable.
  • Impurity importances are fast but biased; permutation importance on held-out data is more trustworthy and model-agnostic.
  • Partial dependence and ICE show how predictions change with a feature.
  • Importance is not causation, and a suspiciously strong feature often signals leakage.

Next: Probability Calibration and Decision Thresholds