Explanation Max-Sensitivity

Description

Explanation max-sensitivity measures how stable an explanation is: whether inputs that differ only slightly, in ways that should not change the answer, receive very different explanations. For each input, the evaluator samples many nearby points within a small radius (for example, no feature moved by more than 0.02 on a 0 to 1 scale), recomputes the explanation at each with the same explanation method, and records the largest distance between a recomputed explanation and the original; Yeh et al. (2019) defined the measure and estimated it by random sampling. Dividing each change in the explanation by the distance the input moved, and taking the largest ratio, gives the closely related local Lipschitz estimate of Alvarez-Melis and Jaakkola (2018), which they maximised with an optimiser rather than by sampling; the two are not interchangeable, because the ratio is dominated by the smallest input changes and can be unbounded where max-sensitivity is finite, so they can rank methods differently. Lower scores mean a more stable explanation, and scores are compared across explanation methods, or across settings of one method, on the same inputs, radius and distance measure. The test treats the explanation method as a black box, so it applies to attribution scores from any method, but on its own it says nothing about faithfulness: a constant explanation that ignores the model is perfectly stable.

Example Use Cases

Explainability

A consumer lender tells declined applicants the main reasons for each decision, taken from Shapley-value attributions of its credit model. Its model-risk team measures max-sensitivity on a sample of declined applications, perturbing features only within bounds the lending policy treats as immaterial, to choose between two sampling budgets for the explainer, because applicants who differ trivially should not be given different main reasons.

An Earth-observation team comparing attribution methods for a multi-label land-cover classifier on satellite image patches computes max-sensitivity for each method together with a most-relevant-first deletion curve, and short-lists methods that do well on both. Pairing the two matters because smoothing an explanation lowers its sensitivity, and a stable method is worth choosing only if it also tracks the model.

A vehicle perception team averages gradient heatmaps over noisy copies of each camera frame to steady them for safety reviewers. It sets the noise level by tracking max-sensitivity and infidelity together on validation frames as the level rises, and picks the point where sensitivity has levelled off before infidelity starts to increase, since too much smoothing produces stable maps that no longer reflect the detector.

Limitations

A low score does not mean the explanation is right. The most stable explanation possible is a constant one that ignores the model, and heavy smoothing keeps lowering sensitivity while eventually making the explanation less faithful. Max-sensitivity should be reported with a measure of faithfulness on the same inputs, and a method should not be chosen for stability alone.
The sampled maximum is a lower bound on the true worst case: with a handful of random samples per input (Captum's default is 10), the largest change found can be far below the largest that exists. Deliberately chosen perturbations that leave the prediction unchanged can alter explanations dramatically, so a low score does not show that explanations resist manipulation.
The number depends on settings that differ between tools: the radius, the distance measure for inputs and explanations, the number of samples and whether explanations are normalised first. By default Captum samples 10 points within a radius of 0.02 and Quantus 200 points within 0.2, ten times wider, while Yeh et al. used 50 points within 0.1 and normalised explanations to unit length. Scores are comparable only under identical settings and identically scaled inputs.
A high score can come from the model rather than the explanation method. Near a decision boundary, or where the model's output changes quickly, a faithful explanation should change, and Yeh et al. note that explanations built from the model's own sensitivity can be more sensitive than the model. Perturbations that change the prediction should be excluded or reported separately.
The test assumes inputs can be nudged by small continuous amounts. For words, categories or other discrete inputs no such small change exists, and the fallback of comparing explanations of similar examples already in the test data is weaker and depends on how many close neighbours the data contain. The per-unit Lipschitz form can be unbounded for gradient explanations of ReLU networks, one reason Yeh et al. prefer the undivided maximum, which stays finite.

Resources

Research Papers

On the (In)fidelity and Sensitivity of Explanations

Chih-Kuan Yeh et al.•Jan 1, 2019

We consider objective evaluation measures of saliency explanations for complex black-box machine learning models. We propose simple robust variants of two notions that have been considered in recent literature: (in)fidelity, and sensitivity. We analyze optimal explanations with respect to both these measures, and while the optimal explanation for sensitivity is a vacuous constant explanation, the optimal explanation for infidelity is a novel combination of two popular explanation methods. By varying the perturbation distribution that defines infidelity, we obtain novel explanations by optimizing infidelity, which we show to out-perform existing explanations in both quantitative and qualitative measurements. Another salient question given these measures is how to modify any given explanation to have better values with respect to these measures. We propose a simple modification based on lowering sensitivity, and moreover show that when done appropriately, we could simultaneously improve both sensitivity as well as fidelity.

On the Robustness of Interpretability Methods

David Alvarez-Melis and Tommi S. Jaakkola•Jun 21, 2018

We argue that robustness of explanations---i.e., that similar inputs should give rise to similar explanations---is a key desideratum for interpretability. We introduce metrics to quantify robustness and demonstrate that current methods do not perform well according to these metrics. Finally, we propose ways that robustness can be enforced on existing interpretability approaches.

Software Packages

captum

Aug 27, 2019

Model interpretability and understanding for PyTorch

Tutorials

Getting started — Quantus documentation

Documentations

Metrics — Captum API reference

Model Interpretability for PyTorch

Related Techniques

Name	Description	Assurance Goals
Explanation Infidelity	Explanation infidelity scores a feature-attribution explanation (one importance score per input feature) by how well it predicts what the model does when the input is changed. The evaluator chooses a distribution of meaningful perturbations, for example small random noise around a reference input, or setting random subsets of features or square patches of an image to a reference value, and draws many perturbations from it. For each one it compares the change in output that the explanation predicts, the sum over features of each feature's perturbation multiplied by its attribution score, with the change the model actually shows, its output on the original input minus its output on the perturbed one. Infidelity is the average squared difference between the two, estimated by random sampling, so lower is better; Yeh et al. (2019), who proposed it, showed that several common explanation methods each have the lowest infidelity for one particular perturbation distribution, so the distribution chosen defines what counts as faithful in a given test. Scores are averaged over a set of inputs and compared across explanation methods, or across settings of one method, under the same perturbations; unlike a deletion curve, the score uses the size of each attribution as well as its rank.	Explainability
Explainer Latency Benchmarking	Explainer latency benchmarking measures how long an explanation method takes to produce an explanation for a single prediction, on the hardware, software and settings it will run with, and compares the result with the time that the decision it supports allows. The evaluator first sets the time budget, the share of the time the decision allows that is left for the explanation once the prediction is ready, then runs the explainer exactly as it will be deployed (the same model, background data, number of samples and batch size) on a representative set of real inputs, timing each explanation from request to a result ready to display, after warm-up runs and with one-off set-up costs, such as building the explainer, recorded separately. Because the time varies with the model, the data, the hardware and the load on the system, each configuration is timed many times, as machine-learning inference benchmarks such as MLPerf Inference time a model, under realistic concurrent demand where the explainer runs as a shared service, and the result is reported as a distribution: the median, high percentiles such as the 95th or 99th, the slowest case and the share of explanations over budget. Canha et al.'s benchmark framework for explanation methods, which scores methods on properties measurable without human participants, includes this property under the name speed and rates it on a five-level scale with cut-offs at 0.1, 1, 5 and 10 seconds aligned with human response-time limits; the scale suits comparisons between methods, while a claim about a deployment needs the measured percentiles set against that deployment's own budget. The output is a latency profile for each explainer configuration, recorded with its hardware, software versions and load, which shows whether explanations will arrive in time and lets a team choose between explainers and settings on speed alongside the explanation quality measured by other tests.	Explainability
Deletion and Insertion Curves	Deletion and insertion curves test whether a feature-attribution explanation, such as a heatmap over an image or highlighted words in a text, puts first the input features the model actually relies on. For the deletion curve, the evaluator removes features in the order the explanation ranks them, most important first, replacing each with a neutral value such as zero, the dataset mean, random noise or a blurred copy, and records the model's score for the explained class after each step; a faithful ranking makes the score fall quickly, so a smaller area under the curve is better (equivalently, a larger area over it, the AOPC score Samek et al. report). The insertion curve runs the other way: it starts from an input with every feature removed or blurred, restores features in ranked order, and rewards a fast rise. For text, the same procedure is often reported as comprehensiveness (how far the predicted-class probability drops when the top-ranked words are deleted) and sufficiency (how far it drops when only those words are kept), either at one cut-off or averaged over cut-offs from the top 1% to the top 50% of words. Explanation methods are compared on the same model, inputs and replacement value, against a random ranking as a baseline, and the result shows how well the ranking tracks the model's behaviour, not whether the highlighted features make sense to a person.	Explainability