Explanation Infidelity

Description

Explanation infidelity scores a feature-attribution explanation (one importance score per input feature) by how well it predicts what the model does when the input is changed. The evaluator chooses a distribution of meaningful perturbations, for example small random noise around a reference input, or setting random subsets of features or square patches of an image to a reference value, and draws many perturbations from it. For each one it compares the change in output that the explanation predicts, the sum over features of each feature's perturbation multiplied by its attribution score, with the change the model actually shows, its output on the original input minus its output on the perturbed one. Infidelity is the average squared difference between the two, estimated by random sampling, so lower is better; Yeh et al. (2019), who proposed it, showed that several common explanation methods each have the lowest infidelity for one particular perturbation distribution, so the distribution chosen defines what counts as faithful in a given test. Scores are averaged over a set of inputs and compared across explanation methods, or across settings of one method, under the same perturbations; unlike a deletion curve, the score uses the size of each attribution as well as its rank.

Example Use Cases

Explainability

A bank's model-risk team must choose which attribution method supplies the reason scores shown to credit analysts for a neural-network default model. It computes infidelity on held-out applications for three candidate methods, with perturbations that set random subsets of features to a documented reference applicant and with rescaling switched on, and records the perturbation distribution in the validation report beside the method it selects.

A manufacturer's quality team uses a convolutional network to flag defects in weld radiographs and smooths its gradient heatmaps by averaging them over noisy copies of each image. It sets the amount of noise by computing infidelity under square-patch perturbations sized to typical defects on validation radiographs and choosing the level with the lowest score, since in the original study infidelity first fell and then rose as smoothing increased.

A hospital's clinical AI team retrains its deterioration-risk model each quarter and shows ward staff the attribution scores behind each alert. After every retraining it recomputes rescaled infidelity on the latest validation data, with the perturbations fixed at first validation, for the deployed attribution settings and two alternatives, and holds the release for review if the deployed settings are no longer the lowest, since the absolute score also moves with the retrained model.

Limitations

What counts as faithful is set by the perturbation distribution. Yeh et al. show that several common explanations each have the lowest possible infidelity for one particular distribution (for example, Shapley values for the feature-subset scheme that SHAP uses), so a comparison can be won by choosing matching perturbations. The distribution should be fixed in advance and justified by the question the explanation is meant to answer.
The score is a squared error between predicted and actual changes in output, so it depends on the scale of the attribution scores: two methods that rank features identically but on different scales get different infidelity. Yeh et al. rescale each explanation optimally before scoring, and Captum offers this rescaling as an option that is off by default; without it, comparisons across methods mislead.
Some attribution methods multiply each feature's sensitivity by the input's distance from a reference and others do not, and the perturbation term has to be computed to match. Yeh et al. compare the two kinds separately, and Captum asks the user to state which kind is being scored; getting this wrong produces a score that reflects the mismatch rather than the explanation.
It follows from the definition, though Yeh et al. do not discuss it, that the predicted change is a sum of per-feature effects. Where the model's response to the chosen perturbations is not additive, as when pixels within an object or words within a phrase interact, no attribution can reach zero infidelity. The floor is the same for every method scored on the same model and perturbations, so rankings stand, but the score is not the explanation's error alone and cannot be compared across models.
The score is an average over random perturbations, each needing a model evaluation. Yeh et al. used 1,000 perturbations per input; Captum's default is 10. Few samples give noisy scores, so small differences between methods need repeated runs, while many samples multiply the cost by the inputs, methods and settings compared. Quantus's built-in version perturbs square patches and so handles image data only.

Resources

Research Papers

On the (In)fidelity and Sensitivity of Explanations

Chih-Kuan Yeh et al.•Jan 1, 2019

We consider objective evaluation measures of saliency explanations for complex black-box machine learning models. We propose simple robust variants of two notions that have been considered in recent literature: (in)fidelity, and sensitivity. We analyze optimal explanations with respect to both these measures, and while the optimal explanation for sensitivity is a vacuous constant explanation, the optimal explanation for infidelity is a novel combination of two popular explanation methods. By varying the perturbation distribution that defines infidelity, we obtain novel explanations by optimizing infidelity, which we show to out-perform existing explanations in both quantitative and qualitative measurements. Another salient question given these measures is how to modify any given explanation to have better values with respect to these measures. We propose a simple modification based on lowering sensitivity, and moreover show that when done appropriately, we could simultaneously improve both sensitivity as well as fidelity.

Software Packages

captum

Aug 27, 2019

Model interpretability and understanding for PyTorch

Tutorials

ImageNet Example All Metrics (Quantus tutorial)

[JMLR 2023] Quantus is an eXplainable AI toolkit for responsible evaluation of neural network explanations - understandable-machine-intelligence-lab/Quantus

Documentations

Metrics — Captum API reference

Model Interpretability for PyTorch

Related Techniques

Name	Description	Assurance Goals
Explanation Max-Sensitivity	Explanation max-sensitivity measures how stable an explanation is: whether inputs that differ only slightly, in ways that should not change the answer, receive very different explanations. For each input, the evaluator samples many nearby points within a small radius (for example, no feature moved by more than 0.02 on a 0 to 1 scale), recomputes the explanation at each with the same explanation method, and records the largest distance between a recomputed explanation and the original; Yeh et al. (2019) defined the measure and estimated it by random sampling. Dividing each change in the explanation by the distance the input moved, and taking the largest ratio, gives the closely related local Lipschitz estimate of Alvarez-Melis and Jaakkola (2018), which they maximised with an optimiser rather than by sampling; the two are not interchangeable, because the ratio is dominated by the smallest input changes and can be unbounded where max-sensitivity is finite, so they can rank methods differently. Lower scores mean a more stable explanation, and scores are compared across explanation methods, or across settings of one method, on the same inputs, radius and distance measure. The test treats the explanation method as a black box, so it applies to attribution scores from any method, but on its own it says nothing about faithfulness: a constant explanation that ignores the model is perfectly stable.	Explainability
Deletion and Insertion Curves	Deletion and insertion curves test whether a feature-attribution explanation, such as a heatmap over an image or highlighted words in a text, puts first the input features the model actually relies on. For the deletion curve, the evaluator removes features in the order the explanation ranks them, most important first, replacing each with a neutral value such as zero, the dataset mean, random noise or a blurred copy, and records the model's score for the explained class after each step; a faithful ranking makes the score fall quickly, so a smaller area under the curve is better (equivalently, a larger area over it, the AOPC score Samek et al. report). The insertion curve runs the other way: it starts from an input with every feature removed or blurred, restores features in ranked order, and rewards a fast rise. For text, the same procedure is often reported as comprehensiveness (how far the predicted-class probability drops when the top-ranked words are deleted) and sufficiency (how far it drops when only those words are kept), either at one cut-off or averaged over cut-offs from the top 1% to the top 50% of words. Explanation methods are compared on the same model, inputs and replacement value, against a random ranking as a baseline, and the result shows how well the ranking tracks the model's behaviour, not whether the highlighted features make sense to a person.	Explainability
Explainer Latency Benchmarking	Explainer latency benchmarking measures how long an explanation method takes to produce an explanation for a single prediction, on the hardware, software and settings it will run with, and compares the result with the time that the decision it supports allows. The evaluator first sets the time budget, the share of the time the decision allows that is left for the explanation once the prediction is ready, then runs the explainer exactly as it will be deployed (the same model, background data, number of samples and batch size) on a representative set of real inputs, timing each explanation from request to a result ready to display, after warm-up runs and with one-off set-up costs, such as building the explainer, recorded separately. Because the time varies with the model, the data, the hardware and the load on the system, each configuration is timed many times, as machine-learning inference benchmarks such as MLPerf Inference time a model, under realistic concurrent demand where the explainer runs as a shared service, and the result is reported as a distribution: the median, high percentiles such as the 95th or 99th, the slowest case and the share of explanations over budget. Canha et al.'s benchmark framework for explanation methods, which scores methods on properties measurable without human participants, includes this property under the name speed and rates it on a five-level scale with cut-offs at 0.1, 1, 5 and 10 seconds aligned with human response-time limits; the scale suits comparisons between methods, while a claim about a deployment needs the measured percentiles set against that deployment's own budget. The output is a latency profile for each explainer configuration, recorded with its hardware, software versions and load, which shows whether explanations will arrive in time and lets a team choose between explainers and settings on speed alongside the explanation quality measured by other tests.	Explainability