Cross-validation
Description
Cross-validation estimates how well a model will perform on data it has not seen, and how much that estimate depends on the particular data used. The dataset is divided into k parts called folds, the model is trained on all but one fold and scored on the held-out fold, and this repeats until every fold has served as the held-out set once. The scores are averaged to give a performance estimate, and their spread gives a rough indication of how much the result depends on which data the model was trained on. Common variants are stratified cross-validation, where each fold keeps the overall class proportions, and leave-one-out, where each fold is a single example. A large gap between training scores and held-out scores points to overfitting, where a model has learned quirks of its training data that do not carry over to new data.
Example Use Cases
Reliability
A hospital data-science team has records for about 900 patients and is building a readmission-risk model. It runs 10-fold cross-validation with each patient's admissions kept in the same fold, then reports the mean and spread of held-out scores. A wide spread, or a large gap to the training score, tells the team the model is not yet stable enough to take to a prospective trial.
A bank's fraud team compares three candidate classifiers by running the same stratified 10-fold cross-validation on a year of labelled transactions, so every candidate is scored on identical folds and the rare fraud cases are spread evenly across them. They select the model with the best mean recall at a fixed false-alarm rate and drop any candidate whose fold-to-fold scores vary widely.
A materials-science group has only 120 labelled samples from costly experiments. Instead of trusting one train-test split, it repeats 5-fold cross-validation with different random splits and reports the spread of the accuracy estimate, which decides whether the model is reliable enough to guide the next round of experiments.
Limitations
- The number of folds trades cost against bias and variance. Each fold is another training run. Few folds (2 to 5) train on less data and tend to understate performance, whereas leave-one-out has little bias but unstable estimates and the highest cost. Ten folds is a common compromise, not a rule.
- Assumes records are independent. If the same patient, customer or site appears in several records, a fold can be trained on records nearly identical to those it is tested on, and the score is optimistic. Folds must then be built by group, not drawn at random.
- Ignores time order. Training on later records and testing on earlier ones lets future information leak into the estimate and flatters the model. For time-ordered data, use an evaluation that always tests on data later than the training data, such as rolling-origin evaluation.
- Anything learned from the data must be fitted inside each training fold, including scaling, feature selection and hyperparameter tuning. Done once on the full dataset beforehand, it leaks test-fold information and inflates the scores. Tuning and reporting on the same folds is also optimistic, so nested cross-validation is needed for both.
- A score from a small sample is imprecise, and the spread across folds understates the true uncertainty. In one neuroimaging study, error bars on a cross-validated score were roughly plus or minus 10% with 100 samples, so a small gain between two models can be noise.
Resources
Research Papers
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Cross-Validatory Choice and Assessment of Statistical Predictions
A generalized form of the cross-validation criterion is applied to the choice and assessment of prediction using the data-analytic concept of a prescription. The examples used to illustrate the application are drawn from the problem areas of univariate estimation, linear regression and analysis of variance.