Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 40 ยท Feature Importance and SHAP

Permutation Importance

Measure how much a trained model really relies on each feature by shuffling it on held-out data. Includes repeats, correlated features, grouped permutation and limitations.

Built-in importances look at how the trees were built. Permutation importance asks a more direct question: if this feature became useless, how much worse would the model's predictions get? You answer it by shuffling one column of a held-out dataset, which destroys its relationship with the target, and measuring the drop in the score. The idea works for any model, and it fixes the training-data and high-cardinality problems from the previous lesson.

The algorithm

  1. Score the trained model on a validation set (the baseline).
  2. For each feature: shuffle that column, score again, and record baseline - shuffled_score.
  3. Repeat the shuffle several times and average, because one shuffle is noisy.
  4. A big drop means the model relies on the feature. A drop near zero means the model ignores it. A negative drop means the feature was slightly harmful on this data.
flowchart LR
  A["Validation data"] --> B["Baseline score"]
  A --> C["Shuffle one column"]
  C --> D["Score again"]
  B --> E["Importance = baseline - shuffled"]
  D --> E

The model is not retrained: only the input column changes.

With scikit-learn

sklearn.inspection.permutation_importance works with the scikit-learn wrappers of both libraries. Here a noise column has many unique values, the same trap that fooled the built-in importance:

Python Permutation importance on held-out data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The built-in total gain ranks the two noise columns above the useful binary feature. The permutation drop, measured on validation data, puts the two useful features clearly on top (about 0.13 and 0.08 AUC) and the noise columns at about -0.005, which is zero within the standard deviation of 0.005 to 0.006. Always look at the standard deviation: a "drop" smaller than twice its standard deviation is not a finding.

Use a metric that matches your goal. For ranking problems use roc_auc or average_precision; for regression use neg_root_mean_squared_error; for probability quality use neg_log_loss. Different scorers can rank features differently.

Correlated features

Permutation importance has its own weak spot. If two columns carry the same information, shuffling one of them hardly hurts, because the model can still read the other. Both features then look unimportant, even though the signal is vital.

Python Two copies of one signal hide each other

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the copy present, the original shows a drop of about 0.29 and the copy only about 0.03, and the pair as a whole looks less important than it is: without the copy, signal alone shows about 0.36. The model can read either column, so shuffling one of them hurts less. With several near-copies the effect is stronger and every member can look unimportant. The fix is grouped permutation: shuffle a whole group of related columns together (the same row permutation for all of them) and report one importance for the group. You can write it in a few lines:

Python
def group_importance(model, X, y, groups, scorer, repeats=10, seed=0):
    rng = np.random.default_rng(seed)
    base = scorer(model, X, y)
    out = {}
    for name, cols in groups.items():
        drops = []
        for _ in range(repeats):
            Xp = X.copy()
            idx = rng.permutation(len(X))
            Xp[cols] = X[cols].to_numpy()[idx]
            drops.append(base - scorer(model, Xp, y))
        out[name] = (np.mean(drops), np.std(drops))
    return out

Practical rules

  • Always use held-out data, never the training set. On training data, memorized features look important.
  • Repeat the shuffle (5 to 30 times) and look at the spread.
  • Mind the cost. The cost is features x repeats predictions on the validation set. For hundreds of features, use a sample of rows or only the top candidates from gain.
  • Be careful with unrealistic combinations. Shuffling creates rows that never occur in reality (for example, an age of 5 with a salary of 200,000), and the model is asked to predict on them. Strongly dependent features make the numbers harder to trust.
  • Use it for feature selection. Drop features with a drop of zero or below, retrain, and confirm the validation score does not fall.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In permutation importance, what is shuffled and what is retrained?

    The trained model stays fixed; only one column of the evaluation data is shuffled and the score drop is measured.
  2. 2. Why should permutation importance be computed on validation data instead of training data?

    On training data, features used to memorize noise appear important. Held-out data shows what generalizes.
  3. 3. Several nearly identical features each show a tiny permutation drop. What is the likely explanation, and the fix?

    When shuffling one copy the model still reads the other, so both seem unimportant. Shuffling the pair together reveals the true importance.
  4. 4. A feature has a mean drop of 0.002 with a standard deviation of 0.004 over repeats. How should you read it?

    The drop is smaller than its spread, so the evidence for any effect is weak.

Summary

  • Permutation importance shuffles a column of held-out data and measures how much the score falls; the model is not retrained.
  • It avoids the training-data and cardinality biases of gain and weight, and works with any metric and model.
  • Repeat shuffles and report the spread.
  • Correlated features hide each other; permute them as a group.
  • It shows reliance, not direction. For per-prediction explanations, continue with SHAP.

Next: SHAP Values for Tree Models