Partial Dependence Plots
Description
Partial Dependence Plots show how changing one or two features affects a model's predictions on average. The technique works by varying the selected feature(s) across their full range whilst keeping all other features fixed at their original values, then averaging the predictions. This creates a clear visualisation of whether increasing or decreasing a feature tends to increase or decrease predictions, and reveals patterns like linear trends, plateaus, or threshold effects that help explain model behaviour.
Example Use Cases
Explainability
Analysing how house prices change with property size in a real estate prediction model, revealing whether the relationship is linear or if there are diminishing returns for very large properties.
Examining how customer age affects predicted loan default probability in a credit scoring model, showing whether risk increases steadily with age or has specific age ranges with higher risk.
Visualising how temperature affects crop yield predictions in agricultural models, identifying optimal temperature ranges and potential threshold effects.
Limitations
- Assumes the plotted features are independent of the others. When they are correlated, the averaging pairs values that never occur together, so the curve partly reflects how the model behaves in regions with no data.
- Shows only the average effect. If a feature acts in opposite directions for different groups of instances, the effects cancel and the curve can look flat, so interactions with other features stay hidden.
- Limited to one or two features at a time, because higher-dimensional plots cannot be read. Effects that depend on three or more features together cannot be shown.
- The basic method needs a pass over the whole dataset for every grid value, so cost grows with dataset size, the number of grid points and the time the model takes to predict.
Resources
Research Papers
Greedy function approximation: A gradient boosting machine.
Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent “boosting” paradigm is developed for additive expansions based on any fitting criterion.Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such “TreeBoost” models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.