Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 76 ยท Modelling

Explaining Model Behaviour

Look inside any model: read coefficients with care, use permutation importance and partial dependence, and find weak spots with residual and segment error analysis.

A model that predicts well is useful. A model you can also explain is trusted, debugged and improved. Stakeholders ask "why did it predict this?" and "what matters most?"; a data scientist asks "where is it wrong, and why?". These questions are not decoration: explanations catch bugs (a feature that should not matter shows up as the most important one), reveal leakage, expose unfair behaviour, and suggest the next feature to build.

This lesson works with a small housing dataset where we know how prices are really generated, so we can check each explanation against the truth. We compare a linear regression, which is easy to read, with a gradient-boosting model, which is more accurate but opaque. The tools shown (coefficients, permutation importance, partial dependence, residual analysis and error analysis by segment) work for almost any model. If you need the basics of the models themselves, read Linear Regression for Inference and Prediction and Decision Trees, Random Forests and Boosting.

flowchart TD
    Q["What do you want to know?"] --> A["How does the model<br/>use each feature overall?"]
    Q --> B["How does the prediction<br/>change as one feature changes?"]
    Q --> C["Where and why does<br/>the model make errors?"]
    A --> A1["Coefficients (linear)<br/>Permutation importance (any)"]
    B --> B1["Partial dependence<br/>and ICE curves"]
    C --> C1["Residual plots<br/>Error by segment"]

Match the tool to the question you are asking.

The data and two models

The synthetic table describes 1,200 homes: floor area, number of rooms, age of the building, distance to the centre (km) and the zone (centre, suburb or rural). Behind the scenes the price (in thousands) depends on area, with a bigger payoff per square metre in the centre; distance reduces it; zone adds a premium; and age has a U-shaped effect (new buildings and historic ones are worth more than middle-aged ones). Rooms matter little once area is known, because area and rooms are strongly related.

We fit a linear regression and a HistGradientBoostingRegressor on 70% of the rows and score both on the other 30%. Zone is one-hot encoded, with the centre as the reference category.

Python Two models, and what the coefficients say

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Boosting is more accurate (a mean absolute error of about 11.9 against 16.4), which is the usual trade: accuracy against transparency. The linear model's coefficients are readable: each extra square metre adds about 1.96 (thousand) to the price, each kilometre from the centre subtracts about 3.0, and suburb homes cost about 114 less than centre homes with the same other features (the zone coefficients are large because the centre is the reference and, in the data, centre homes also gain more per square metre than the single area coefficient can express).

Reading coefficients carefully

Coefficients are the most natural explanation, but they are easy to over-read:

  • Units. A raw coefficient depends on how the feature is measured. The coefficient of area (1.96) looks tiny beside zone_rural (-138) only because area is measured in square metres and the zone dummy jumps from 0 to 1. Standardized coefficients (the "per 1 std dev" column) put features on a common scale: area (56) and the zone columns (about 57 to 58 in absolute value) are the big drivers, distance is a clear second tier (12), and rooms and age are small.
  • Linearity. The coefficient of age is about 0.02: the linear model says age does not matter. In truth, age matters a lot, but its effect is U-shaped, and a straight line through a U is flat. A coefficient near zero means "no linear effect", not "no effect".
  • Correlated features. Area and rooms move together, so the model can split credit between them in arbitrary ways. Their individual coefficients are unstable even when predictions are fine.
  • Causation. A coefficient describes association in this data. It does not say that building an extra room raises the price by that amount.

Permutation importance

Permutation importance works for any model. Take the test set, shuffle one column (breaking its link to the target), predict again and see how much the error grows. A feature the model relies on causes a big increase; a feature it ignores causes none. Repeat the shuffle several times to see the spread.

Compute it on held-out data, because on training data a model may lean on features that only help it memorize. Use the metric you care about (here the mean absolute error, so the importance is "how many thousand of error this feature saves").

Python Permutation importance for both models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the boosting model, area and the two zone columns dominate (permuting area adds about 54 to the error), followed by age (8, so the model has found the effect the linear coefficient missed) and distance (6). Rooms barely matter once area is known (0.6). Note that the zone is spread over two dummy columns, and each one is important because shuffling either hides the difference between the centre and the rest. For the linear model the importance of age is almost zero (0.04), consistent with its flat coefficient.

The last block shows the main weakness. When two columns carry the same information, shuffling one leaves the other to cover for it, so both look less important than they really are: here area drops from about 54 to 34, and the near-copy gets 11. Before interpreting importances, remove or combine near-duplicate features, or group them and permute the group together.

Importance says how much the model uses a feature, not how much the feature matters in the real world, and not whether the relationship is causal. A model can rely heavily on a proxy or on a leaked column.

Partial dependence and ICE curves

Importance says whether a feature matters. Partial dependence (PD) shows how: it plots the average prediction as one feature sweeps across its range, with the other features left as they are in the data. ICE (individual conditional expectation) curves draw one line per row, so you can see whether the average hides different behaviour between homes. If the ICE lines are roughly parallel, the average tells the story; if they cross or fan out, the feature interacts with others.

Python Partial dependence and ICE for age, distance and area

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The age curve is a clear U, falling from the newest homes to a minimum around the middle of the age range and rising again for the oldest. That is the truth we built in and exactly what the linear coefficient hid. The distance curve falls steadily, and the area curve rises. The area ICE lines fan out: the effect of an extra square metre is not the same for every home, because in the centre it is worth more. A single average line would hide that, which is why ICE is worth plotting.

PD assumes you can move one feature while holding the rest fixed. For strongly correlated features (area and rooms) this creates unrealistic combinations, such as a 40 square metre home with seven rooms, and the curve can mislead. Read it as "what the model does", not "what the world does".

Residual analysis

A residual is the actual value minus the prediction. If the model captured the structure, residuals look like patternless noise around zero. Any pattern in them points to something the model missed. Plot residuals against the predictions and against each important feature, and compare errors across groups.

Python Residual plots and error by segment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The left plot shows the linear model's U-shaped residuals: it overestimates mid-aged homes and underestimates the newest and oldest ones (the age-band table makes the same point with numbers: an average residual of about +13 for the newest homes, -13 and -12 for the middle bands and +10 for the oldest). That is a model specification problem, and it has a cheap fix: add an age squared feature to the linear model. The boosting residuals stay within about 3 of zero in every band.

Error analysis by segment

Overall error hides unequal performance. Group the test rows by meaningful segments (zone, size band, time period, customer type) and compare errors. A model with a fine average can be poor for one group, and that group may be the one the business cares about, or one that deserves fair treatment (see Ethics, Fairness and Responsible Data Use).

In the example, the linear model is clearly worst in the centre (mean absolute error 21.2 against 14.6 in rural and 15.5 in suburb zones), the segment where the true price depends on an interaction between zone and area that a purely additive model cannot represent. The boosting model is better everywhere (13.1 in the centre), though centre homes are still the hardest because they have the highest and most varied prices. Next steps would be adding a zone-by-area feature to the linear model, or collecting more centre examples.

A practical workflow for error analysis:

  1. Compute errors on held-out data, never on training data.
  2. Slice by each important feature and by business segments, and look for groups with large or one-sided (biased) errors.
  3. Read the biggest individual misses. Are they data errors, rare cases, or a missing feature?
  4. Decide on an action: fix data, add a feature, change the model, or document a known limitation.

What these tools do not do

Explanations are summaries of model behaviour on a given dataset. They can be unstable (a small data change can reorder importances), they can disagree with each other, and they never prove causation. Use them to build understanding and to find problems, and combine them with domain knowledge. Two further tools worth knowing, though not run here, are SHAP values (a per-prediction attribution for any model, from a separate package) and simple surrogate models (a small decision tree trained to mimic a complex model).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A linear regression gives the coefficient of a feature as about 0, yet the feature clearly matters in the data. What is a likely explanation?

    A coefficient measures a linear association. For a U-shaped effect the best straight line is nearly flat, so the coefficient is near zero even though the feature matters.
  2. 2. Why compute permutation importance on the test set rather than the training set?

    Importance should reflect the features that help predict unseen data. Training data can reward memorization.
  3. 3. Two features are near-copies of each other. What happens to their permutation importances?

    Shuffling one column leaves the twin available, so the error rises less. Combine or drop near-duplicates before interpreting.
  4. 4. ICE lines for a feature fan out and cross. What does this suggest?

    If the effect of a feature differs between rows, the individual curves differ, and the average curve can hide it.
  5. 5. A residual plot against a feature shows a clear curve instead of random scatter around zero. What should you conclude?

    Residuals should look like noise. A visible pattern means some structure remains unexplained, so add a transform, interaction or use a more flexible model.

Summary

  • Explanations answer different questions: overall use (coefficients, permutation importance), shape of the effect (partial dependence, ICE) and failures (residuals, segment errors).
  • Coefficients depend on units, only capture linear effects, and are unstable with correlated features; standardize to compare them.
  • Permutation importance works for any model; compute it on held-out data and beware of near-duplicate features.
  • Partial dependence and ICE curves show how predictions change with a feature; the average can hide interactions.
  • Residuals should look like noise; patterns point to missing features or the wrong model form.
  • Compare errors across segments: a good average can hide a group the model serves poorly.

Next: Reproducible Analysis, the habits that make all of this work repeatable.