Jackknife Resampling

Description

Jackknife resampling, also called leave-one-out resampling, shows how much a result computed from a dataset depends on any single record. The result can be a model's coefficients, its predictions or a performance score; it is recomputed n times on a dataset of n records, leaving out a different record each time and refitting the model where the result depends on one. The spread of the recomputed results, after a standard rescaling, estimates the bias and variance of the original result, meaning how far it would move with a slightly different sample. Records whose removal moves the result most are flagged as influential, and may be outliers or data errors worth checking. The method assumes the records are independent and that no single record changes the result abruptly.

Example Use Cases

Reliability

A hospital data team fits a readmission-risk model on 600 patient records and refits it 600 times, leaving out one patient each time. The spread of the refitted coefficients shows which risk factors are stable enough to report, and the records whose removal shifts the fit most go back to clinicians to check for entry errors.

A water utility's analysts estimate how rainfall drives reservoir inflow from 40 annual observations. Leaving out each year in turn gives a variance estimate for the fitted slope and shows whether a single flood year drives the result, before the figure goes into a planning document.

Transparency

A credit-risk team documents its scorecard's stability for model validators by refitting it with each of 800 sampled loan records left out. It reports the records whose removal changes any coefficient by more than a stated tolerance, so validators can see how much of the model rests on a few loans.

Limitations

Computationally heavy when the result needs a model refit, since n records mean n refits, which is impractical for datasets with thousands or millions of records or for costly training. Simple statistics that are cheap to recompute do not have this problem.
Works only for statistics that change smoothly when one record is removed. For quantities that jump, such as the number of clusters in a mixture model, leave-one-out estimates are unreliable. Even for smooth statistics the variance estimate tends to run high, so intervals are conservative.
Assumes the records are independent. With repeated or clustered measurements, such as several visits per patient or neighbouring regions, leaving out one record leaves its close relatives in the data, so the refits barely change and the spread understates the true uncertainty.
Provides limited insight into model behaviour on truly novel data, as each jackknife sample is only minimally different from the full training set.
Measures how sensitive a result is to the sample in hand, not whether the model is right. A stable fit can still be systematically wrong, so a narrow spread across refits is no evidence of accuracy, or of performance on a different population.

Resources

Research Papers

Notes on bias in estimation

M. H. QUENOUILLE•Jan 1, 1956

1. One of the commonest problems in statistics is, given a series of observations Xj, xit..., xn, to find a function of these, tn(xltxit..., xn), which should provide an estimate of an unknown parameter 0. The desirable properties of estimation procedures have been discussed fully elsewhere. They are: (a) That the estimator should be efficient according to some definition of efficiency pre-viously arranged. Most commonly, the reciprocal of the variance of the estimates is taken as a measure of its efficiency, as this is most useful where central limit theory may be relevant. (6) That the estimator should utilize all the information contained in the observations, x1,xi,...,xn concerning the parameter 6. This is not always possible, but, if such an estimator exists, it is called sufficient. (c) That the estimator should be consistent, i.e. tn converges in some probabilistic sense to 6, usually lim tn-* • 6. n-»-a> (d) That the estimator should be unbiased, i.e. E(tn) = 6. The method of maximum likelihood is popular in that it satisfies properties (a) to (c), whence, by evaluating E(tn), an unbiased statistic may be derived. That such evaluation is necessary is obvious when it is remembered that i/r(tn) is, by the same theory, the estimator of ft(6), and, since in general E[fr(tn)] ^ iJr[E(tn)'], it will be the exception rather than the rule for a maximum-likelihood estimator to be unbiased. Provided the exceptions may be simply evaluated no real difficulty arises. However, often the complexity of the evaluation presents a major drawback and some simple approach is then desirable. 2. If the observations are taken in random order, the estimator tn may often be written where klt kit..., km are unbiased estimates of the cumulants K1,K2I...,

Software Packages

scikit-learn

scikit-learn: machine learning in Python. Contribute to scikit-learn/scikit-learn development by creating an account on GitHub.

Tutorials

The jackknife (lecture notes, Statistics 490, Carnegie Mellon University)

Documentations

scikit-learn model_selection.LeaveOneOut

Scikit-learn Developers•Jan 1, 2007

Related Techniques

Name	Description	Assurance Goals
Bootstrap Ensembles	Bootstrap ensembles estimate how much a model's predictions depend on the particular sample of data it was trained on. The training set is resampled with replacement many times, each resample (a bootstrap replicate) holding as many cases as the original so that some cases are repeated and others left out; one model is trained on each replicate with the same settings, and every new input is passed through all of the models. The spread of the models' predictions for an input, reported as a standard deviation or as a range between percentiles, estimates the uncertainty in the model's prediction for that input, and inputs where the models disagree are those the training data constrain least. The spread covers uncertainty in the fitted model only, not the random variation in individual outcomes, so an interval meant to contain a case's actual outcome must add an estimate of that noise, for example from the models' errors on held-out data. Putting a confidence interval around a measured performance score is a different procedure, which resamples the evaluation set and retrains nothing; see Bootstrap Confidence Intervals for Evaluation Metrics.	Reliability Transparency Fairness
Deep Ensembles	Deep ensembles estimate how uncertain a neural network's predictions are by training several copies of the same network independently and comparing what they predict. Each copy starts from different random initial weights, sees the training data in a different random order, and is trained to output a full predictive distribution, such as a probability for each class or a predicted value with its spread, rather than a single answer. The ensemble's prediction is the average of its members. Where members agree the prediction is usually more reliable, though agreement alone is no guarantee, and where they disagree the model is showing uncertainty about which answer is right, known as epistemic or model uncertainty. In the original study, ensembles of about five networks were better calibrated than single networks, meaning their stated confidence matched how often they were right, and expressed higher uncertainty on inputs from classes absent from the training data.	Reliability Transparency
Knowledge Boundary Probing	Knowledge boundary probing maps what a language model can and cannot reliably answer. A probe set is built from questions across the topics the system will face, including some with known answers and some that cannot be answered. Each question is asked in several phrasings, because a model may answer one wording correctly and fail on another. Responses are scored for correctness, for whether the model signals doubt when it should, and for whether its stated or computed confidence tracks whether it is right. The result is a topic-by-topic map of what the model knows, knows only under some phrasings, and does not know, which informs where to deploy it, what to tell users about its limits, and where to add training data or human review.	Transparency Reliability