Mean Decrease Impurity
Description
Mean Decrease Impurity (MDI) scores how much each feature contributes to a tree-based model, such as a random forest, by looking at how the model's decision trees split the training data. Each split sends cases into two groups, and a good split makes each group more uniform in its outcome, which is a drop in impurity, measured with a score such as Gini impurity. For each feature, MDI adds up the drops from every split that uses it, weights each by the share of training cases reaching that split, and averages over all trees. The scores are usually scaled to sum to one, so they show relative importance. Because they come from the training data, they describe how the model was built, not how well each feature predicts on new data.
Example Use Cases
Explainability
Determining the most influential genetic markers in a decision tree model predicting disease susceptibility, by identifying which markers consistently lead to the purest splits between healthy and diseased patient groups.
A model-validation team reviews a gradient-boosted loan-default model and ranks its inputs by MDI before sign-off. They investigate any identifier-like or many-valued field, such as customer ID or postcode, that ranks near the top, because MDI favours such fields and they may reflect overfitting rather than real predictive signal.
An environmental analytics team fits a random forest to forecast air quality and uses MDI to choose which weather stations' readings to drop from a simpler model. They then confirm on held-out data that accuracy holds, because MDI alone reflects the training data.
Limitations
- MDI favours features with many distinct values, because more possible split points give more chances of a good split by chance. Strobl and colleagues found that uninformative continuous or many-category variables were preferred over informative ones when predictors differed in type or number of categories.
- Scores are calculated from the training data, so a feature can rank high because the model fitted noise in it. A high score does not show that the feature helps predictions on unseen data, and scikit-learn recommends permutation importance on held-out data as the alternative.
- MDI gives only a non-negative score for each feature. It does not show whether higher values of a feature raise or lower the prediction, so it cannot support a claim about the direction of a feature's effect.
- Scores vary from tree to tree, and the scikit-learn example plots this spread as error bars. A forest with few trees, or a small training set, can therefore give a ranking that shifts when the model is refitted.
- Correlated predictors compete for splits. One can mask another's importance, and changing the fitted ensemble or candidate-feature settings can change their relative ranks.
Resources
Research Papers
Bias in random forest variable importance measures: Illustrations, sources and a solution
BACKGROUND: Variable importance measures for random forests have been receiving increased attention as a means of variable selection in many classification tasks in bioinformatics and related scientific fields, for instance to select a subset of genetic markers relevant for the prediction of a certain disease. We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories. This is particularly important in genomics and computational biology, where predictors often include variables of different types, for example when predictors include both sequence data and continuous variables such as folding energy, or when amino acid sequence data show different numbers of categories. RESULTS: Simulation studies are presented illustrating that, when random forest variable importance measures are used with data of varying types, the results are misleading because suboptimal predictor variables may be artificially preferred in variable selection. The two mechanisms underlying this deficiency are biased variable selection in the individual classification trees used to build the random forest on one hand, and effects induced by bootstrap sampling with replacement on the other hand. CONCLUSION: We propose to employ an alternative implementation of random forests, that provides unbiased variable selection in the individual classification trees. When this method is applied using subsampling without replacement, the resulting variable importance measures can be used reliably for variable selection even in situations where the potential predictor variables vary in their scale of measurement or their number of categories. The usage of both random forest algorithms and their variable importance measures in the R system for statistical computing is illustrated and documented thoroughly in an application re-analyzing data from a study on RNA editing. Therefore the suggested method can be applied straightforwardly by scientists in bioinformatics research.
Understanding variable importances in forests of randomized trees
Despite growing interest and practical use in various scientific areas, variable importances derived from tree-based ensemble methods are not well understood from a theoretical point of view. In this work we characterize the Mean Decrease Impurity (MDI) variable importances as measured by an ensemble of totally randomized trees in asymptotic sample and ensemble size conditions. We derive a three-level decomposition of the information jointly provided by all input variables about the output in terms of i) the MDI importance of each input variable, ii) the degree of interaction of a given input variable with the other input variables, iii) the different interaction terms of a given degree. We then show that this MDI importance of a variable is equal to zero if and only if the variable is irrelevant and that the MDI importance of a relevant variable is invariant with respect to the removal or the addition of irrelevant variables. We illustrate these properties on a simple example and discuss how they may change in the case of non-totally randomized trees such as Random Forests and Extra-Trees.