Forward Simulation Test

Description

A forward simulation test checks whether people who have studied a model's explanations can predict the model's output for inputs they have not seen, so it tests with people the understanding an explanation method is assumed to give, where fidelity checks only compare the explanation with the model by computation. Doshi-Velez and Kim proposed it as a human-grounded evaluation of interpretability: participants are given an explanation and an input and must state what the model will output, whether or not that output is correct. In the controlled design of Hase and Bansal, participants study a set of examples with the model's outputs, predict the outputs for a separate set of test inputs, then study the same examples with explanations added and predict the same test inputs again, and the gain in accuracy is credited to the explanations (a separate group that studies the examples without explanations guards against a practice gain); the test inputs are kept apart from the explained examples so that explanations cannot give the answers away, and are chosen so that the model is right and wrong equally often, so that predicting the true label scores no better than chance. A counterfactual variant, in Hase and Bansal's form, shows an input, the model's output and its explanation, and asks what the model will output for an edited version of that input, with edits chosen so that half change the output. The result is the change in the share of correct predictions, with a confidence interval, for each explanation method or design compared; it is evidence that people can anticipate the model on the kind of input tested, not that they make better decisions with it.

Example Use Cases

Explainability

A bank's model validation team must choose between feature-attribution bar charts and a short rule list as the explanation shown to credit analysts for a loan-default model. It runs a forward simulation test with 30 credit analysts on past applications, balanced between correct and incorrect model predictions, and adopts the rule list only if its gain over the no-explanation baseline is clearly larger than that of the bar charts; otherwise it treats the two as equal.

An online platform's trust and safety team uses the counterfactual variant to test whether moderators who see which words a toxicity classifier weighted can predict whether it will still flag a comment after one of those words is replaced. A score no better than the baseline tells the team that the highlights do not show how the classifier responds to edits, before it writes moderation guidance that relies on them.

A research team choosing between counterfactual images and saliency heatmaps for a chest X-ray pneumonia classifier shows each group of participants explained example images, then asks them to predict the classifier's decision on 12 other images, three from each cell of its confusion matrix, without explanations. It takes forward to a study with clinicians the explanation whose group predicts best, measured against a group that saw the examples without explanations.

Transparency

A local authority plans to show caseworkers, who are outside the team that built it, the reasons behind a housing-benefit fraud-risk score. Before releasing the reasons in that form, it runs a forward simulation test with caseworkers on unseen cases, half of them cases the score got wrong, using the reasons exactly as they will appear, and revises them if caseworkers who studied them predict the flag no better than caseworkers who did not.

Limitations

A pass shows that people can anticipate the model, not that they decide better with it. Buçinca et al. found that results on a task of predicting an AI's decisions did not predict results on the real decision task, and Poursabzi-Sangdeh et al. found that people who could better simulate a simple, clear model did not follow it more when that helped, and in some experiments were less able to correct its large mistakes. A claim about decisions needs a decision study.
The test is easily made trivial. If the explained examples are reused as test items, explanations that show the predicted class or a probability give the answer away, and if the model is mostly right, participants can score well by predicting the true label. Hase and Bansal found earlier studies that reused the same items for explanation and prediction; results from such designs overstate what the explanation conveys.
Effects are often small against the variation between people and items. With 32 trained undergraduates and about 180 to 320 responses per condition, Hase and Bansal's 95% confidence intervals on the change in accuracy were about 8 to 13 percentage points either side, and only LIME on tabular data clearly improved both forward and counterfactual simulation. A study must be sized for the difference it needs to detect, with a bootstrap over both participants and items.
The score holds for the people tested. Many studies use students or crowd workers because domain experts are costly to recruit, which Doshi-Velez and Kim give as the reason this kind of evaluation is cheaper; a result from lay participants does not show that the clinicians, analysts or caseworkers who will use the explanation can predict the model.
The before-and-after design credits any gain to the explanations, but participants see the same examples and test items twice, and Hase and Bansal had no group that repeated both phases without explanations, so a practice gain is not ruled out. Their participants also found it hard to remember the explained examples while predicting, so the number of examples and the time allowed must be fixed and reported when methods are compared.

Resources

Research Papers

Towards A Rigorous Science of Interpretable Machine Learning

Finale Doshi-Velez and Been Kim•Mar 2, 2017

As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.

Software Packages

InterpretableNLP-ACL2020: code for 'Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?'

Mar 31, 2026

Code for "Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?"

Related Techniques

Name	Description	Assurance Goals
Think-Aloud Study of Explanation Use	A think-aloud study of explanation use asks a small number of participants to say what they are thinking while they use an AI system's explanations to do a realistic task, so that the design team can see how the explanation is read, used, ignored or misread. It applies the think-aloud method from cognitive psychology, which rests on Ericsson and Simon's account of verbal reports: people can voice what they are attending to as they work without changing their thinking, provided they are not asked to report things they would not otherwise attend to, such as the reasons for their actions. In a session, a facilitator gives a short practice task, then sets tasks in which the participant sees the model's output and explanation and makes a decision, and while the participant works gives only neutral reminders, such as 'what are you thinking?', when they fall silent; audio and screen are recorded. The recordings are transcribed and coded, often by thematic analysis, into findings such as parts of the explanation that were misread, parts never looked at and questions it left unanswered, and the team changes the design and tests it again. The output is qualitative, design-stage evidence about how an explanation is understood: it can show that a misreading occurs, but not how often it occurs among users or whether their decisions improve.	Explainability
Explanation Satisfaction Scale	The Explanation Satisfaction Scale is a short questionnaire on which people who have used an AI system's explanations rate how satisfying and sufficient they found them; Hoffman, Mueller, Klein and Litman developed it in the DARPA Explainable AI programme to capture the user's judgement of an explanation, as distinct from a researcher's judgement of its quality. Each item is a statement about the explanation, for example that from it the user understands how the system works, or that it is satisfying, has sufficient detail, seems complete, tells the user how to use the system, is useful to their goals or shows how accurate the system is, and each is rated on a five-point scale from 'I agree strongly' to 'I disagree strongly'. Participants complete it after they have worked with the system and received explanations, usually once for each explanation condition in a study that compares conditions; the published form sets no scoring rule or threshold, and the implementation by one of its authors in the OpenScales library scores it as the mean of the item ratings. Versions differ: the authors' 2018 report gives eight items, their 2023 journal article seven (it drops the item on knowing when to trust the system), and the form on the XAITK site nine (it adds an item on reliability), with a note that researchers may use a subset. The score measures how users judge an explanation, not whether they understood it or decide better with it, and the authors present it beside separate measures of mental models, trust and task performance.	Explainability
Appropriate Reliance Measurement	Appropriate reliance measurement tests whether people who receive an AI system's advice follow it when it is right and keep their own judgement when it is wrong, and whether an explanation, or another aid such as a confidence score, moves them towards that pattern. It uses a two-step decision study on a classification task, the only kind Schemmer et al. define it for: for each task item the participant first decides alone, then sees the AI's advice, with or without the explanation, and makes a final decision, on a set of items with known correct answers drawn so that the AI is wrong on a known share of them. Only items on which the first decision and the advice disagree are counted, because when they agree the final decision cannot show which source the person relied on. Schemmer et al. define two rates over these items: relative AI reliance, the share of items with a wrong first decision and correct advice on which the person switched to the advice, and relative self-reliance, the share of items with a correct first decision and wrong advice on which the person kept their own answer. Comparing the two rates between an explanation condition and a control condition shows whether the explanation reduces under-reliance (rejecting correct advice), reduces over-reliance (accepting wrong advice) or merely raises agreement with the AI, which overall accuracy or agreement with the AI does not show on its own.	Explainability Safety