Deletion and Insertion Curves

Description

Deletion and insertion curves test whether a feature-attribution explanation, such as a heatmap over an image or highlighted words in a text, puts first the input features the model actually relies on. For the deletion curve, the evaluator removes features in the order the explanation ranks them, most important first, replacing each with a neutral value such as zero, the dataset mean, random noise or a blurred copy, and records the model's score for the explained class after each step; a faithful ranking makes the score fall quickly, so a smaller area under the curve is better (equivalently, a larger area over it, the AOPC score Samek et al. report). The insertion curve runs the other way: it starts from an input with every feature removed or blurred, restores features in ranked order, and rewards a fast rise. For text, the same procedure is often reported as comprehensiveness (how far the predicted-class probability drops when the top-ranked words are deleted) and sufficiency (how far it drops when only those words are kept), either at one cut-off or averaged over cut-offs from the top 1% to the top 50% of words. Explanation methods are compared on the same model, inputs and replacement value, against a random ranking as a baseline, and the result shows how well the ranking tracks the model's behaviour, not whether the highlighted features make sense to a person.

Example Use Cases

Explainability

A hospital's cardiology informatics team has a deep network that measures intervals and amplitudes from 12-lead ECGs and must choose which heatmap method to show cardiologists. It perturbs the highest-ranked sample points first, one per step, tracks the rise in measurement error over several thousand ECGs, and compares each method's area under the error curve with that of random heatmaps. Because expert preferences and the curves can disagree, it reports both before choosing.

A regulator's data team routes consumer complaints with a fine-tuned transformer classifier and wants to highlight for caseworkers the words behind each routing decision. Before choosing between two word-attribution methods, it computes comprehensiveness and sufficiency on a validation sample at the 1, 5, 10, 20 and 50% cut-offs and adopts a method only if it clearly beats randomly ranked words on both measures.

A rail operator's engineering team uses an image classifier to flag defects in track inspection photographs and wants heatmaps that point inspectors to the defect. It runs deletion with pixels set to a constant and insertion from a blurred image for three heatmap methods, then repeats the deletion run with a second replacement value, and adopts a method only if it ranks first under both.

Limitations

Removing pixels or words produces inputs unlike anything the model was trained on, so part of the drop in score can be the model reacting to an unfamiliar input rather than to the loss of the information the explanation pointed to, and the shape of the removed region can itself carry class information. Results are least trustworthy when large parts of the input are removed; variants that retrain the model or fill removed pixels from their neighbours reduce the effect.
The score depends on how features are removed and in what order. On one image classifier, the best-ranked method changed when removed pixels were set to the dataset mean instead of random colours, removing the least relevant pixels first ranked the methods roughly in reverse of removing the most relevant first, and rankings varied from image to image. A comparison holds only for the stated replacement value, order, step size and fraction removed, and should be checked under a second replacement.
Only the order of the attribution scores counts, not their size, so explanations with very different magnitudes can receive the same score. Taking absolute values of signed attributions changes the order and hence the result. The curves cannot tell an explanation that ranks the right features but gives them misleading magnitudes from one that gets both right; that needs a measure that uses the attribution values.
The curves show whether the ranking tracks the model, not whether the explanation is useful or correct for the domain: a model that relies on a spurious cue receives a faithful explanation of that cue. In a study of ECG models, ECG experts' choice of the best heatmap method disagreed considerably with the perturbation ranking, so a good curve does not show that experts will find the explanation helpful.
Each curve needs one model evaluation per step, for every input and method compared, so fine-grained curves over large images, long documents or big test sets are expensive. In the original region-perturbation study, two settings over 5,040 images each, with 100 steps repeated ten times because the replacement values were random, took roughly 36 hours on a workstation with 20 hyper-threaded Xeon cores in 2015. Coarser steps cut the cost but make the curve less able to separate methods.

Resources

Research Papers

Evaluating the Visualization of What a Deep Neural Network Has Learned

Wojciech Samek et al.•Nov 1, 2017

RISE: Randomized Input Sampling for Explanation of Black-box Models

Vitali Petsiuk, Abir Das, and Kate Saenko•Jan 1, 2018

ERASER: A Benchmark to Evaluate Rationalized NLP Models

Jay DeYoung et al.•Jan 1, 2020

Software Packages

Xplique (deel-ai/xplique)

Oct 9, 2026

👋 Xplique is a Neural Networks Explainability Toolbox

Tutorials

Attribution Metrics tutorial (Xplique)

Documentations

Deletion — Xplique documentation

Related Techniques

Name	Description	Assurance Goals
Sanity Checks for Saliency Maps	Sanity checks for saliency maps are two randomisation tests that show whether an explanation method's output depends on what a neural network has learned, or would look much the same for a network that had learned nothing. In the model parameter randomisation test, the evaluator recomputes explanations for a fixed set of inputs after replacing the trained weights with random ones, either cumulatively from the output layer down towards the input (cascading randomisation) or one layer at a time with the others left trained (independent randomisation). In the data randomisation test, the evaluator trains a copy of the same architecture on the training data with the labels shuffled, until it has memorised them, and compares its explanations with those of the model trained on the true labels. Similarity between original and randomised explanations is measured with rank correlation, with and without absolute values, and with image similarity scores such as the structural similarity index; a method whose maps stay similar fails, because its output cannot reflect the learned weights or the relationship between inputs and labels, as Guided Backpropagation and Guided GradCAM failed in the original study when the upper layers were randomised, while plain gradients and GradCAM passed. Passing is a necessary condition, not proof of a good explanation: it shows a method responds to the model, not that its explanations are faithful or useful.	Explainability
Explanation Infidelity	Explanation infidelity scores a feature-attribution explanation (one importance score per input feature) by how well it predicts what the model does when the input is changed. The evaluator chooses a distribution of meaningful perturbations, for example small random noise around a reference input, or setting random subsets of features or square patches of an image to a reference value, and draws many perturbations from it. For each one it compares the change in output that the explanation predicts, the sum over features of each feature's perturbation multiplied by its attribution score, with the change the model actually shows, its output on the original input minus its output on the perturbed one. Infidelity is the average squared difference between the two, estimated by random sampling, so lower is better; Yeh et al. (2019), who proposed it, showed that several common explanation methods each have the lowest infidelity for one particular perturbation distribution, so the distribution chosen defines what counts as faithful in a given test. Scores are averaged over a set of inputs and compared across explanation methods, or across settings of one method, under the same perturbations; unlike a deletion curve, the score uses the size of each attribution as well as its rank.	Explainability
Explanation Max-Sensitivity	Explanation max-sensitivity measures how stable an explanation is: whether inputs that differ only slightly, in ways that should not change the answer, receive very different explanations. For each input, the evaluator samples many nearby points within a small radius (for example, no feature moved by more than 0.02 on a 0 to 1 scale), recomputes the explanation at each with the same explanation method, and records the largest distance between a recomputed explanation and the original; Yeh et al. (2019) defined the measure and estimated it by random sampling. Dividing each change in the explanation by the distance the input moved, and taking the largest ratio, gives the closely related local Lipschitz estimate of Alvarez-Melis and Jaakkola (2018), which they maximised with an optimiser rather than by sampling; the two are not interchangeable, because the ratio is dominated by the smallest input changes and can be unbounded where max-sensitivity is finite, so they can rank methods differently. Lower scores mean a more stable explanation, and scores are compared across explanation methods, or across settings of one method, on the same inputs, radius and distance measure. The test treats the explanation method as a black box, so it applies to attribution scores from any method, but on its own it says nothing about faithfulness: a constant explanation that ignores the model is perfectly stable.	Explainability