Think-Aloud Study of Explanation Use

Description

A think-aloud study of explanation use asks a small number of participants to say what they are thinking while they use an AI system's explanations to do a realistic task, so that the design team can see how the explanation is read, used, ignored or misread. It applies the think-aloud method from cognitive psychology, which rests on Ericsson and Simon's account of verbal reports: people can voice what they are attending to as they work without changing their thinking, provided they are not asked to report things they would not otherwise attend to, such as the reasons for their actions. In a session, a facilitator gives a short practice task, then sets tasks in which the participant sees the model's output and explanation and makes a decision, and while the participant works gives only neutral reminders, such as 'what are you thinking?', when they fall silent; audio and screen are recorded. The recordings are transcribed and coded, often by thematic analysis, into findings such as parts of the explanation that were misread, parts never looked at and questions it left unanswered, and the team changes the design and tests it again. The output is qualitative, design-stage evidence about how an explanation is understood: it can show that a misreading occurs, but not how often it occurs among users or whether their decisions improve.

Example Use Cases

Explainability

A team building a nurse-facing explanation for a patient deterioration score runs think-aloud sessions with six ward nurses on a clickable prototype using past patient records. It finds that nurses read a factor that lowered the score as a sign that the patient was improving, relabels the chart, and re-tests the new labels in a second round before fixing the design for a quantitative evaluation.

The developer of a vocal emotion-recognition app that highlights which parts of a recording drove each prediction runs remote think-aloud sessions with a dozen users. One participant reads the highlight colour bar as a trace of the speaker's tone of voice, a misreading that no rating of the explanation would have revealed, and the team changes how the highlight is drawn.

A bank's model risk team preparing to give loan officers counterfactual explanations (for example, the income at which a refused application would have been approved) runs think-aloud sessions with five officers on past applications, including some the model got wrong. It learns which suggestions officers would pass on to applicants and which they reject as implausible, and decides whether to filter the suggestions before release.

Limitations

How participants are asked to talk changes what is observed. A meta-analysis of 94 studies by Fox, Ericsson and Best found that plain thinking aloud left accuracy unchanged, but that asking people to explain or describe their thinking changed performance, and that verbal reports of every kind tended to make tasks take longer. Facilitators must avoid 'why' prompts, and task times from these sessions cannot serve as performance measures.
Talking through a task can make people attend to an explanation more than they would at work. In Buçinca et al.'s think-aloud study, participants' preferences between two explanation types matched the results of a proxy task rather than those of the realistic task, and the authors suggest thinking aloud pushed people into analytical thinking, so these sessions may overstate how closely users read explanations in normal use.
Studies use few participants (between 4 and 14 in the sources for this record), so a finding shows that a problem exists, not how common it is, and cannot support a claim about the share of users who understand an explanation or about decision quality. The sessions also show only the situations their tasks include: tasks need some wrong model outputs if the team wants to see how people use an explanation when the model errs.
Not everything people do with an explanation gets said. Quick, perceptual uses leave little trace: in Zhang and Lim's study, participants called the audio examples intuitive but did not say why they found them simple. People also tend to say what they think the right answer is, the social desirability bias the GOV.UK guidance warns of. Silence about part of an explanation does not mean it went unused or was understood, and a screen recording or gaze data is needed to tell these apart.
Findings depend on the analyst: coding transcripts into themes is interpretive, so a study meant to support an assurance claim needs a stated coding scheme and a second coder. The voice and screen recordings are personal data that need participants' consent and controlled storage, and recordings made on real cases may also hold personal data about the people those cases concern.

Resources

Research Papers

Verbal reports as data.

K. Anders Ericsson and Herbert A. Simon•May 1, 1980

Tutorials

Think aloud study: qualitative studies

How to use a think aloud study to evaluate your digital health product.

Documentations

Thinking Aloud: The #1 Usability Tool

Simple usability tests where users think out loud are cheap, robust, flexible, and easy to learn. Thinking aloud should be the first tool in your UX toolbox, even though it entails some risks and doesn't solve all problems.

Related Techniques

Name	Description	Assurance Goals
Forward Simulation Test	A forward simulation test checks whether people who have studied a model's explanations can predict the model's output for inputs they have not seen, so it tests with people the understanding an explanation method is assumed to give, where fidelity checks only compare the explanation with the model by computation. Doshi-Velez and Kim proposed it as a human-grounded evaluation of interpretability: participants are given an explanation and an input and must state what the model will output, whether or not that output is correct. In the controlled design of Hase and Bansal, participants study a set of examples with the model's outputs, predict the outputs for a separate set of test inputs, then study the same examples with explanations added and predict the same test inputs again, and the gain in accuracy is credited to the explanations (a separate group that studies the examples without explanations guards against a practice gain); the test inputs are kept apart from the explained examples so that explanations cannot give the answers away, and are chosen so that the model is right and wrong equally often, so that predicting the true label scores no better than chance. A counterfactual variant, in Hase and Bansal's form, shows an input, the model's output and its explanation, and asks what the model will output for an edited version of that input, with edits chosen so that half change the output. The result is the change in the share of correct predictions, with a confidence interval, for each explanation method or design compared; it is evidence that people can anticipate the model on the kind of input tested, not that they make better decisions with it.	Explainability Transparency
Explanation Satisfaction Scale	The Explanation Satisfaction Scale is a short questionnaire on which people who have used an AI system's explanations rate how satisfying and sufficient they found them; Hoffman, Mueller, Klein and Litman developed it in the DARPA Explainable AI programme to capture the user's judgement of an explanation, as distinct from a researcher's judgement of its quality. Each item is a statement about the explanation, for example that from it the user understands how the system works, or that it is satisfying, has sufficient detail, seems complete, tells the user how to use the system, is useful to their goals or shows how accurate the system is, and each is rated on a five-point scale from 'I agree strongly' to 'I disagree strongly'. Participants complete it after they have worked with the system and received explanations, usually once for each explanation condition in a study that compares conditions; the published form sets no scoring rule or threshold, and the implementation by one of its authors in the OpenScales library scores it as the mean of the item ratings. Versions differ: the authors' 2018 report gives eight items, their 2023 journal article seven (it drops the item on knowing when to trust the system), and the form on the XAITK site nine (it adds an item on reliability), with a note that researchers may use a subset. The score measures how users judge an explanation, not whether they understood it or decide better with it, and the authors present it beside separate measures of mental models, trust and task performance.	Explainability
Appropriate Reliance Measurement	Appropriate reliance measurement tests whether people who receive an AI system's advice follow it when it is right and keep their own judgement when it is wrong, and whether an explanation, or another aid such as a confidence score, moves them towards that pattern. It uses a two-step decision study on a classification task, the only kind Schemmer et al. define it for: for each task item the participant first decides alone, then sees the AI's advice, with or without the explanation, and makes a final decision, on a set of items with known correct answers drawn so that the AI is wrong on a known share of them. Only items on which the first decision and the advice disagree are counted, because when they agree the final decision cannot show which source the person relied on. Schemmer et al. define two rates over these items: relative AI reliance, the share of items with a wrong first decision and correct advice on which the person switched to the advice, and relative self-reliance, the share of items with a correct first decision and wrong advice on which the person kept their own answer. Comparing the two rates between an explanation condition and a control condition shows whether the explanation reduces under-reliance (rejecting correct advice), reduces over-reliance (accepting wrong advice) or merely raises agreement with the AI, which overall accuracy or agreement with the AI does not show on its own.	Explainability Safety