Ridge Regression Surrogates
Description
A ridge regression surrogate approximates a complex model with a linear equation fitted to the complex model's own predictions. The complex model is run on a set of inputs, and ridge regression, a linear model with a penalty that keeps coefficients stable when features are correlated, is fitted to reproduce its outputs. Each coefficient then shows roughly how far a feature moves the complex model's output, averaged over those inputs. Fidelity, often reported as R-squared (the share of the model's variation the surrogate reproduces on held-out inputs), shows how far the coefficients can be trusted. They describe the complex model, not the real world.
Example Use Cases
Explainability
A model-risk team at a bank fits a ridge surrogate to a gradient-boosted credit-scoring model's scores on a held-out sample, reports how closely the surrogate reproduces them, and uses the standardised coefficients to brief the model-risk committee on which inputs drive scores overall.
A hospital analytics team fits a ridge surrogate to a neural network that predicts readmission risk from tabular records, to check that the features driving the network's predictions overall, such as age and prior admissions, are plausible to clinicians. The findings concern the network, not what causes readmission.
An energy company's forecasting team fits a ridge surrogate to a neural-network load forecaster using a year of hourly weather and calendar inputs, and reads the standardised coefficients to confirm that temperature and time of day dominate. A low R-squared during heatwaves leads the team to record that the surrogate does not describe the model in that regime.
Transparency
An insurer's compliance team gives a regulator a ridge surrogate of its pricing model together with held-out R-squared and a table of each feature's contribution to a sample of quotes, so the regulator can check the stated drivers against the model's actual outputs.
Limitations
- A linear surrogate cannot represent interactions or thresholds. Where the complex model relies on them, the surrogate fits its predictions poorly and its coefficients describe a different model. Report held-out R-squared and treat a low value as a failed surrogate.
- A good average fit can hide poor fit in parts of the input space, such as rare groups or extreme values, so the coefficients may be wrong for exactly the cases that matter. Check fidelity within subgroups and near decision boundaries.
- There is no agreed fidelity level above which a surrogate can be trusted. Whether 50%, 80% or 99% of variance explained is enough depends on the use, so the reviewer has to set and justify a cut-off.
- Ridge shrinks every coefficient and shares weight between correlated features. Coefficient sizes depend on the penalty strength and on feature scaling, so compare them only after standardising the features, and report the penalty used.
- The surrogate explains the complex model's behaviour on the sampled inputs, not the real-world link between features and outcomes. Reading its coefficients as causes, or as evidence about the data, will mislead.
Resources
Research Papers
Extracting Tree-Structured Representations of Trained Networks
A significant limitation of neural networks is that the represen(cid:173) tations they learn are usually incomprehensible to humans. We present a novel algorithm , TREPAN, for extracting comprehensible, symbolic representations from trained neural networks. Our algo(cid:173) rithm uses queries to induce a decision tree that approximates the concept represented by a given network. Our experiments demon(cid:173) strate that TREPAN is able to produce decision trees that maintain a high level of fidelity to their respective networks while being com(cid:173) prehensible and accurate. Unlike previous work in this area, our algorithm is general in its applicability and scales well to large net(cid:173) works and problems with high-dimensional input spaces.