Appropriate Reliance Measurement

Description

Appropriate reliance measurement tests whether people who receive an AI system's advice follow it when it is right and keep their own judgement when it is wrong, and whether an explanation, or another aid such as a confidence score, moves them towards that pattern. It uses a two-step decision study on a classification task, the only kind Schemmer et al. define it for: for each task item the participant first decides alone, then sees the AI's advice, with or without the explanation, and makes a final decision, on a set of items with known correct answers drawn so that the AI is wrong on a known share of them. Only items on which the first decision and the advice disagree are counted, because when they agree the final decision cannot show which source the person relied on. Schemmer et al. define two rates over these items: relative AI reliance, the share of items with a wrong first decision and correct advice on which the person switched to the advice, and relative self-reliance, the share of items with a correct first decision and wrong advice on which the person kept their own answer. Comparing the two rates between an explanation condition and a control condition shows whether the explanation reduces under-reliance (rejecting correct advice), reduces over-reliance (accepting wrong advice) or merely raises agreement with the AI, which overall accuracy or agreement with the AI does not show on its own.

Example Use Cases

Explainability

An insurer considering highlighted claim features as an explanation for its fraud-flagging model runs a two-step study with its claims handlers on 40 past claims, with the model wrong on a quarter of them, comparing handlers who see the flag alone with handlers who also see the highlights. It adopts the highlights only if relative AI reliance rises without relative self-reliance falling, since higher acceptance of all advice, right and wrong, would show deference rather than better judgement.

A conservation charity whose volunteers identify bird species with help from an image classifier measures relative AI reliance and relative self-reliance separately for experienced birders and beginners, and for example-based and written explanations. It chooses which explanation to show each group from the results, because a volunteer's expertise changes how an explanation, especially a wrong one, shifts their reliance on the classifier.

Safety

A hospital whose safety case for a sepsis alert relies on ward doctors overriding wrong alerts, and whose workflow has doctors record their own escalation decision before the alert appears, runs a vignette study in that workflow, with and without the alert's explanation panel. Before switching the panel on, it reports relative self-reliance, and the share of wrong-alert cases that ended in a wrong decision, against the oversight requirement in the safety case.

Limitations

The rates describe behaviour on a task set built to contain many AI errors: Schemmer et al. drew equal numbers of items from each cell of the confusion matrix, which put the AI's accuracy on their 16 items at 50%. The study does not show how people rely on advice when errors are rarer, as they usually are in service, so it supports a claim about the conditions tested, not a direct estimate of over-reliance in deployment.
Only items on which the first decision and the advice disagree enter the two rates, so each rate can rest on a handful of items per person, and items on which the person and the AI are wrong together, the errors that oversight cannot catch, enter neither rate. Plan the sample around the expected number of disagreements, and report beside the rates the share of wrong-advice items that ended in a wrong final decision.
Asking for a decision before the advice is shown is itself an intervention. Guo et al.'s reanalysis of an earlier study found reliance closer to the appropriate level when people decided first than when they saw the AI's advice first, so a two-step study may show better reliance than a deployed system that presents the advice straight away.
The rates judge each decision by whether the advice turned out to be right, not by whether following it was the better bet given what the person could see. Guo et al. argue that this outcome-based definition mixes two failures, wrong beliefs about how often the AI is right and poor use of those beliefs, and treats a close call the same as an obvious one, so a low rate does not say which to fix.
Results depend on who takes part and on how good the explanations are. Morrison et al. found that participants' expertise changed how incorrect explanations affected their reliance, and that example-based explanations misled participants more than written ones, so a result from novices, or from explanations that were always correct, does not carry over to expert users or to explanations that are sometimes wrong.

Resources

Research Papers

Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations

Max Schemmer et al.•Mar 27, 2023

Related Techniques

Name	Description	Assurance Goals
Forward Simulation Test	A forward simulation test checks whether people who have studied a model's explanations can predict the model's output for inputs they have not seen, so it tests with people the understanding an explanation method is assumed to give, where fidelity checks only compare the explanation with the model by computation. Doshi-Velez and Kim proposed it as a human-grounded evaluation of interpretability: participants are given an explanation and an input and must state what the model will output, whether or not that output is correct. In the controlled design of Hase and Bansal, participants study a set of examples with the model's outputs, predict the outputs for a separate set of test inputs, then study the same examples with explanations added and predict the same test inputs again, and the gain in accuracy is credited to the explanations (a separate group that studies the examples without explanations guards against a practice gain); the test inputs are kept apart from the explained examples so that explanations cannot give the answers away, and are chosen so that the model is right and wrong equally often, so that predicting the true label scores no better than chance. A counterfactual variant, in Hase and Bansal's form, shows an input, the model's output and its explanation, and asks what the model will output for an edited version of that input, with edits chosen so that half change the output. The result is the change in the share of correct predictions, with a confidence interval, for each explanation method or design compared; it is evidence that people can anticipate the model on the kind of input tested, not that they make better decisions with it.	Explainability Transparency
Explanation Satisfaction Scale	The Explanation Satisfaction Scale is a short questionnaire on which people who have used an AI system's explanations rate how satisfying and sufficient they found them; Hoffman, Mueller, Klein and Litman developed it in the DARPA Explainable AI programme to capture the user's judgement of an explanation, as distinct from a researcher's judgement of its quality. Each item is a statement about the explanation, for example that from it the user understands how the system works, or that it is satisfying, has sufficient detail, seems complete, tells the user how to use the system, is useful to their goals or shows how accurate the system is, and each is rated on a five-point scale from 'I agree strongly' to 'I disagree strongly'. Participants complete it after they have worked with the system and received explanations, usually once for each explanation condition in a study that compares conditions; the published form sets no scoring rule or threshold, and the implementation by one of its authors in the OpenScales library scores it as the mean of the item ratings. Versions differ: the authors' 2018 report gives eight items, their 2023 journal article seven (it drops the item on knowing when to trust the system), and the form on the XAITK site nine (it adds an item on reliability), with a note that researchers may use a subset. The score measures how users judge an explanation, not whether they understood it or decide better with it, and the authors present it beside separate measures of mental models, trust and task performance.	Explainability
Think-Aloud Study of Explanation Use	A think-aloud study of explanation use asks a small number of participants to say what they are thinking while they use an AI system's explanations to do a realistic task, so that the design team can see how the explanation is read, used, ignored or misread. It applies the think-aloud method from cognitive psychology, which rests on Ericsson and Simon's account of verbal reports: people can voice what they are attending to as they work without changing their thinking, provided they are not asked to report things they would not otherwise attend to, such as the reasons for their actions. In a session, a facilitator gives a short practice task, then sets tasks in which the participant sees the model's output and explanation and makes a decision, and while the participant works gives only neutral reminders, such as 'what are you thinking?', when they fall silent; audio and screen are recorded. The recordings are transcribed and coded, often by thematic analysis, into findings such as parts of the explanation that were misread, parts never looked at and questions it left unanswered, and the team changes the design and tests it again. The output is qualitative, design-stage evidence about how an explanation is understood: it can show that a misreading occurs, but not how often it occurs among users or whether their decisions improve.	Explainability

Appropriate Reliance Measurement

Description

Example Use Cases

Explainability

Safety

Limitations

Resources

Research Papers

Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations

Tags

Explainability Dimensions

Safety Dimensions

Other Categories