Explainer Latency Benchmarking

Description

Explainer latency benchmarking measures how long an explanation method takes to produce an explanation for a single prediction, on the hardware, software and settings it will run with, and compares the result with the time that the decision it supports allows. The evaluator first sets the time budget, the share of the time the decision allows that is left for the explanation once the prediction is ready, then runs the explainer exactly as it will be deployed (the same model, background data, number of samples and batch size) on a representative set of real inputs, timing each explanation from request to a result ready to display, after warm-up runs and with one-off set-up costs, such as building the explainer, recorded separately. Because the time varies with the model, the data, the hardware and the load on the system, each configuration is timed many times, as machine-learning inference benchmarks such as MLPerf Inference time a model, under realistic concurrent demand where the explainer runs as a shared service, and the result is reported as a distribution: the median, high percentiles such as the 95th or 99th, the slowest case and the share of explanations over budget. Canha et al.'s benchmark framework for explanation methods, which scores methods on properties measurable without human participants, includes this property under the name speed and rates it on a five-level scale with cut-offs at 0.1, 1, 5 and 10 seconds aligned with human response-time limits; the scale suits comparisons between methods, while a claim about a deployment needs the measured percentiles set against that deployment's own budget. The output is a latency profile for each explainer configuration, recorded with its hardware, software versions and load, which shows whether explanations will arrive in time and lets a team choose between explainers and settings on speed alongside the explanation quality measured by other tests.

Example Use Cases

Explainability

A card issuer shows its fraud analysts a feature attribution beside each held transaction and needs it within the two seconds before the review queue refreshes. Benchmarking on the production server at peak queue load, it finds that a sampling-based explainer misses the budget for about one explanation in twenty while an explainer specific to its tree ensemble stays well inside it, and adopts the second once its attributions pass the same faithfulness tests.

A hospital adding an explanation panel to a sepsis-risk alert in its electronic health record agrees with clinicians that the panel must load within one second of a nurse opening the alert. The informatics team times explanations on the hospital's own servers with a week of real alerts at typical volume, reports the 95th percentile and the slowest case against that limit, and repeats the measurement after each model or software update.

An online marketplace shows each seller the main reasons its moderation model held a new listing for review, and the page must update within three seconds. Before launch, the team times the service on its production servers with a week of recorded traffic replayed at its real pace; the first request after each quiet period is far slower because the explainer and its background data load on demand. It keeps a loaded explainer running, re-measures, and reports the slowest case.

Limitations

Speed and explanation quality trade off through the explainer's settings. Sampling-based explainers run faster with fewer samples or a smaller background dataset, but SHAP's documentation notes that more samples give lower-variance estimates, so a faster setting gives noisier explanations. A latency result holds only for the settings tested, and must be measured at the settings whose faithfulness and stability were checked, or a configuration may meet the budget with untested explanations.
Times depend on the hardware, the implementation and the load at the moment of measurement, which Canha et al. single out as the main caveat on their speed metric, so a figure from a developer's workstation does not carry over to a shared production server or an edge device. Any change to the model, the explainer version, the hardware or the traffic pattern calls for re-measurement, and the configuration must be recorded with the result for it to be checked or repeated.
Claims about the slowest cases need many timed explanations. MLPerf Inference, which sets tail-latency bounds for model inference in the same way, sizes its runs so that a 99th-percentile bound holds with 99 per cent confidence, which at its chosen margin takes about 263,000 queries. A few hundred timings can support a claim about the median but not about the 99th percentile, and a single timed run supports neither.
A fast explanation is not necessarily a good one. Latency says nothing about whether an explanation reflects what the model did or whether people can use it, and it leaves out the time a person needs to read it. It supports a claim about explanations only alongside tests of faithfulness and stability and, where people act on the explanations, a study of whether they can use them within the time available.
The budget has to come from the decision, not from a generic scale. Canha et al.'s cut-offs reflect general limits on human response time, which may be looser or tighter than a particular workflow allows, and a budget that leaves out the time to run the model, move the data and display the result will be met in the benchmark and missed in use; MLPerf likewise derives its server-scenario latency bounds from the user's overall time target.

Resources

Research Papers

A Functionally-Grounded Benchmark Framework for XAI Methods: Insights and Foundations from a Systematic Literature Review

Dulce Canha et al.•Jul 14, 2025

MLPerf Inference Benchmark

Vijay Janapa Reddi et al.•May 1, 2020

Tutorials

PyTorch Benchmark (PyTorch recipe)

Related Techniques

Name	Description	Assurance Goals
Explanation Infidelity	Explanation infidelity scores a feature-attribution explanation (one importance score per input feature) by how well it predicts what the model does when the input is changed. The evaluator chooses a distribution of meaningful perturbations, for example small random noise around a reference input, or setting random subsets of features or square patches of an image to a reference value, and draws many perturbations from it. For each one it compares the change in output that the explanation predicts, the sum over features of each feature's perturbation multiplied by its attribution score, with the change the model actually shows, its output on the original input minus its output on the perturbed one. Infidelity is the average squared difference between the two, estimated by random sampling, so lower is better; Yeh et al. (2019), who proposed it, showed that several common explanation methods each have the lowest infidelity for one particular perturbation distribution, so the distribution chosen defines what counts as faithful in a given test. Scores are averaged over a set of inputs and compared across explanation methods, or across settings of one method, under the same perturbations; unlike a deletion curve, the score uses the size of each attribution as well as its rank.	Explainability
Explanation Max-Sensitivity	Explanation max-sensitivity measures how stable an explanation is: whether inputs that differ only slightly, in ways that should not change the answer, receive very different explanations. For each input, the evaluator samples many nearby points within a small radius (for example, no feature moved by more than 0.02 on a 0 to 1 scale), recomputes the explanation at each with the same explanation method, and records the largest distance between a recomputed explanation and the original; Yeh et al. (2019) defined the measure and estimated it by random sampling. Dividing each change in the explanation by the distance the input moved, and taking the largest ratio, gives the closely related local Lipschitz estimate of Alvarez-Melis and Jaakkola (2018), which they maximised with an optimiser rather than by sampling; the two are not interchangeable, because the ratio is dominated by the smallest input changes and can be unbounded where max-sensitivity is finite, so they can rank methods differently. Lower scores mean a more stable explanation, and scores are compared across explanation methods, or across settings of one method, on the same inputs, radius and distance measure. The test treats the explanation method as a black box, so it applies to attribution scores from any method, but on its own it says nothing about faithfulness: a constant explanation that ignores the model is perfectly stable.	Explainability
Deletion and Insertion Curves	Deletion and insertion curves test whether a feature-attribution explanation, such as a heatmap over an image or highlighted words in a text, puts first the input features the model actually relies on. For the deletion curve, the evaluator removes features in the order the explanation ranks them, most important first, replacing each with a neutral value such as zero, the dataset mean, random noise or a blurred copy, and records the model's score for the explained class after each step; a faithful ranking makes the score fall quickly, so a smaller area under the curve is better (equivalently, a larger area over it, the AOPC score Samek et al. report). The insertion curve runs the other way: it starts from an input with every feature removed or blurred, restores features in ranked order, and rewards a fast rise. For text, the same procedure is often reported as comprehensiveness (how far the predicted-class probability drops when the top-ranked words are deleted) and sufficiency (how far it drops when only those words are kept), either at one cut-off or averaged over cut-offs from the top 1% to the top 50% of words. Explanation methods are compared on the same model, inputs and replacement value, against a random ranking as a baseline, and the result shows how well the ranking tracks the model's behaviour, not whether the highlighted features make sense to a person.	Explainability