Permutation Tests

Description

A permutation test asks how likely an observed result would be if there were no real relationship in the data. The tester shuffles the data many times in a way that breaks the relationship under test, recalculates the metric each time, and takes the p-value as the share of shuffled results at least as extreme as the real one. Shuffling the outcome labels and retraining tests whether a classifier's accuracy beats chance, and shuffling group membership tests whether a gap between groups in the rate of decisions, such as approvals, is larger than chance would produce. A restricted variant, Ojala and Garriga's second test, shuffles each feature's values separately within each class, which keeps every feature's own spread but breaks the relationships between features, so a classifier that makes significantly fewer errors on the real data than on the shuffled copies is using those relationships. The test needs no assumption about the shape of the data's distribution, but a small p-value does not show that an effect is large, important or causal.

Example Use Cases

Reliability

A hospital research group trains a classifier on 80 patients with thousands of gene-expression measurements. With so few cases, a good cross-validated accuracy could be luck, so it repeats the whole training and testing procedure on thousands of label-shuffled copies and reports the result only if it beats nearly all of them.

A council analytics team replacing a case-prioritisation model checks whether the new model's higher accuracy on the same test cases could be chance. It randomly swaps the two models' predictions case by case 10,000 times, recomputes the gap each time, and switches only if the real gap is larger than nearly all of the shuffled ones.

Fairness

A bank's model-risk team sees a higher approval rate for one applicant group than another in a loan model's decisions. Before escalating, it shuffles group membership across applicants thousands of times and recomputes the gap, to test whether a gap this large could arise by chance. A significant gap prompts a fuller review but does not by itself show unlawful discrimination.

Explainability

A cardiology team using a gradient-boosted model to flag high blood pressure expects height and weight to matter together, not only alone. It shuffles each feature within each outcome class 1,000 times and tests at the 5% level whether the model errs less on the real data, to decide if a simpler model treating features separately would do. A non-significant result means either the features are unrelated within classes or the model ignores the relationship.

Limitations

Each permutation needs the metric recalculated, and a model may need retraining every time, so thousands of permutations of a large model can be expensive. Stricter thresholds need more permutations, because the smallest possible p-value is 1 divided by the number of permutations plus one, and repeated runs can disagree near the threshold.
Assumes exchangeability of observations under the null hypothesis, which may be violated in time series or hierarchical data structures, where simple shuffling gives p-values that are too small or too large.
A significant result shows that the data are unlikely under the stated null, such as 'features and labels are independent'. It does not show that the effect is large, important or causal, and shuffling cannot rule out a confounder or an artefact of how the data were collected.
Testing many features or groups at once produces some false positives by chance, so the p-values need correcting, for example with a false discovery rate procedure. Without that, a feature-by-feature test over many features will flag some that are only noise.
What is shuffled defines the question answered. Shuffling labels tests whether features and labels are related at all, while shuffling features within classes tests whether a classifier uses dependencies between features. The wrong scheme answers a different question from the one the assurance claim needs.

Resources

Research Papers

Significance Tests Which May be Applied to Samples from Any Populations

E. J. G. Pitman•Jan 1, 1937

Permutation Tests for Studying Classifier Performance

Markus Ojala and Gemma C. Garriga•Dec 1, 2009

We explore the framework of permutation-based p-values for assessing the behavior of the classification error. In this paper we study two simple permutation tests. The first test estimates the null distribution by permuting the labels in the data; this has been used extensively in classification problems in computational biology. The second test produces permutations of the features within classes, inspired by restricted randomization techniques traditionally used in statistics. We study the properties of these tests and present an extensive empirical evaluation on real and synthetic data. Our analysis shows that studying the classification error via permutation tests is effective; in particular, the restricted permutation test clearly reveals whether the classifier exploits the interdependency between the features in the data.

Software Packages

scipy/scipy: SciPy library main repository

Oct 9, 2026

SciPy library main repository

Tutorials

Test with permutations the significance of a classification score - scikit-learn

This example demonstrates the use of permutation_test_score to evaluate the significance of a cross-validated score using permutations. Dataset: We will use the Iris plants dataset, which consists ...

Documentations

scipy.stats.permutation_test - SciPy Manual

Related Techniques

Name	Description	Assurance Goals
Permutation Importance	Permutation Importance measures how much a trained model relies on each input feature. For each feature, the values in a dataset are shuffled at random, which breaks the link between that feature and the outcome, and the model is scored again. The fall in score, for example in accuracy, is the feature's importance: a large fall means the model depends on the feature, and little or no fall means it does not. Repeating the shuffle several times shows how much the result varies. The method works with any model, but it shows what the model uses, not what causes the outcome in the world.	Explainability
Bootstrap Confidence Intervals for Evaluation Metrics	Bootstrap confidence intervals put a range around a model's measured performance, showing how much the score could differ had another test set of the same size been drawn from the same population. The labelled held-out evaluation set is resampled with replacement thousands of times, each resample holding as many cases as the original so that some cases appear more than once and others not at all, and the metric (an error rate, the area under a curve, a translation score) is recomputed on each resample from the model's stored predictions, with no retraining. The spread of these recomputed values stands in for the sampling distribution of the metric: the percentile interval takes its 2.5th and 97.5th percentiles for a 95 per cent interval, and the bias-corrected and accelerated (BCa) interval shifts those percentiles to correct for bias and skew in that distribution. The statistic can also be the difference between two models scored on the same cases, recomputed on each shared resample, which shows whether an observed improvement is larger than the noise from test-set sampling. The result is the metric with lower and upper bounds at a stated confidence level, so a requirement such as a maximum false-positive rate can be checked against the bound rather than the point estimate.	Reliability
SHapley Additive exPlanations	SHAP (SHapley Additive exPlanations) explains one prediction by sharing out the gap between that prediction and a baseline, usually the model's average output over a background dataset, among the input features. Each feature receives a signed score, where positive means it pushed the prediction above the baseline and negative means it pushed it below. The scores are Shapley values from game theory, meaning each is the feature's average effect over every order in which the features could be added, and together they add up exactly to the gap. Exact calculation is costly, so tools usually estimate the scores by sampling or use shortcuts for particular model types, such as a fast exact method for tree ensembles, and averaging the absolute scores over many predictions gives a global ranking of features. In a fairness use, the difference in average output between two groups is shared out feature by feature in the same way, showing which inputs account for the gap.	Explainability Fairness