Bootstrap Ensembles
Description
Bootstrap ensembles estimate how much a model's predictions depend on the particular sample of data it was trained on. The training set is resampled with replacement many times, each resample (a bootstrap replicate) holding as many cases as the original so that some cases are repeated and others left out; one model is trained on each replicate with the same settings, and every new input is passed through all of the models. The spread of the models' predictions for an input, reported as a standard deviation or as a range between percentiles, estimates the uncertainty in the model's prediction for that input, and inputs where the models disagree are those the training data constrain least. The spread covers uncertainty in the fitted model only, not the random variation in individual outcomes, so an interval meant to contain a case's actual outcome must add an estimate of that noise, for example from the models' errors on held-out data. Putting a confidence interval around a measured performance score is a different procedure, which resamples the evaluation set and retrains nothing; see Bootstrap Confidence Intervals for Evaluation Metrics.
Example Use Cases
Reliability
An electricity network operator forecasting next-day demand at each substation trains 100 gradient-boosted models on bootstrap replicates of three years of meter readings, resampled in whole-week blocks because consecutive readings are correlated. Planners hold extra reserve at substations where the models' forecasts spread widely, because those are the sites whose forecasts the historical data settle least.
Transparency
A hospital's readmission-risk tool shows clinicians, beside each patient's predicted risk, the range of risks given by models trained on bootstrap replicates of the training cohort, resampling patients rather than admissions so that one patient's admissions stay together, so that a clinician can see when an estimate rests on few comparable past patients and give it less weight in discharge planning.
Fairness
A local authority reviewing a housing-need scoring model compares the average spread of bootstrap-ensemble scores across applicant groups and finds it widest for a group with few past applications, which shows that the model's scores for that group are least settled by the data; it collects more cases before letting the scores inform priority for that group.
Limitations
- Every member of the ensemble is a full training run, and every prediction must be made by every member, so training cost, storage and prediction time all grow in proportion to the number of replicates. For large neural networks this usually limits the ensemble to a handful of members, too few for stable percentiles of the spread.
- The spread measures uncertainty in the fitted model, not the noise in outcomes. Presented as the range within which a case's actual outcome will fall, it understates the uncertainty, most of all where outcomes are noisy and training data plentiful, so the outcome noise has to be estimated separately, for example from held-out errors, and added.
- The spread reflects only what the procedure lets vary: which cases are in each training set and, for models trained from a random start such as neural networks, each member's starting point. Fixed choices such as features, architecture and training settings add no spread. A model consistently wrong for a group the data under-represent can show little spread there, and members can agree closely on inputs unlike the training data, so a small spread is not evidence of reliability.
- Each replicate contains on average only about 63 per cent of the distinct training cases, so each member learns from less information than a model trained on all the data. With small datasets or rare classes, some replicates may contain almost none of a rare class or subgroup, which inflates the spread for those cases and can leave individual members unusable.
- Resampling single cases assumes they are independent. With repeated measurements of the same person, time series or other grouped data, case resampling spreads related cases across replicates and understates the uncertainty, unless whole groups or time blocks are resampled instead.
Resources
Research Papers
Bootstrap Methods: Another Look at the Jackknife
We discuss the following problem: given a random sample X = (X1, X2, …, Xn) from an unknown probability distribution F, estimate the sampling distribution of some prespecified random variable R(X, F), on the basis of the observed data x. (Standard jackknife theory gives an approximate mean and variance in the case R(X, F) = θ(F̂) − θ(F), θ some parameter of interest.) A general method, called the "bootstrap," is introduced, and shown to work satisfactorily on a variety of estimation problems. The jackknife is shown to be a linear approximation method for the bootstrap. The exposition proceeds by a series of examples: variance of the sample median, error rates in a linear discriminant analysis, ratio estimation, estimating regression parameters, etc.
Practical Confidence and Prediction Intervals
We propose a new method to compute prediction intervals. Espe(cid:173) cially for small data sets the width of a prediction interval does not only depend on the variance of the target distribution, but also on the accuracy of our estimator of the mean of the target, i.e., on the width of the confidence interval. The confidence interval follows from the variation in an ensemble of neural networks, each of them trained and stopped on bootstrap replicates of the original data set. A second improvement is the use of the residuals on validation pat(cid:173) terns instead of on training patterns for estimation of the variance of the target distribution. As illustrated on a synthetic example, our method is better than existing methods with regard to extrap(cid:173) olation and interpolation in data regimes with a limited amount of data, and yields prediction intervals which actual confidence levels are closer to the desired confidence levels.