Permutation Importance
Description
Permutation Importance measures how much a trained model relies on each input feature. For each feature, the values in a dataset are shuffled at random, which breaks the link between that feature and the outcome, and the model is scored again. The fall in score, for example in accuracy, is the feature's importance: a large fall means the model depends on the feature, and little or no fall means it does not. Repeating the shuffle several times shows how much the result varies. The method works with any model, but it shows what the model uses, not what causes the outcome in the world.
Example Use Cases
Explainability
Assessing which patient characteristics (e.g., age, blood pressure, cholesterol) matter most to a medical diagnosis model by shuffling each one in turn on held-out patients and recording the drop in performance, so a clinician can judge whether the model relies on clinically relevant factors.
A bank's model-risk team shuffles each input to a card-fraud model in turn on held-out transactions, to check the model leans on transaction behaviour and not on a field that leaks the answer, such as a case-handling code recorded after fraud was confirmed.
A flood-forecasting team shuffles each rainfall, river-level and soil-moisture input of a model trained on historic data, to check that forecasts depend on the physical drivers hydrologists expect and not on a calendar variable.
Limitations
- When features are strongly correlated, shuffling one leaves the model able to rely on its correlated partner, so both can show low importance even though the model uses them. Shuffling also creates feature combinations that do not occur in real data, which can distort the scores.
- Computationally expensive for large datasets or complex models. The model is scored once per feature per shuffle, so cost grows with the number of features, the number of repeats and the size of the evaluation data. No retraining is needed.
- The score for a feature includes the effect of its interactions with other features and cannot separate them from its own effect. If two features interact, shuffling either one lowers the score, so importances need not add up to the model's total performance.
- The ranking can change with the scoring metric, particularly for imbalanced classification, where accuracy and F1-score can give different pictures of which features matter.
- Scores computed on training data can reflect overfitting: a feature the model has memorised can look important even though it does not help on new data. Use held-out data, and repeat the shuffle to see how much the scores vary.
Resources
Research Papers
Random Forests
Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. The generalization error for forests converges a.s. to a limit as the number of trees in the forest becomes large. The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them. Using a random selection of features to split each node yields error rates that compare favorably to Adaboost (Y. Freund & R. Schapire, Machine Learning: Proceedings of the Thirteenth International conference, ***, 148–156), but are more robust with respect to noise. Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the splitting. Internal estimates are also used to measure variable importance. These ideas are also applicable to regression.
All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})=\mathbf{x}^{T}β$ with a fixed coefficient vector $β$) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.