Human-in-the-Loop Safeguards

Description

Human-in-the-loop safeguards hold an AI system's output at a checkpoint until a trained person has reviewed it. Designers set the rules that send a case to review, such as low model confidence (the system's own estimate of how likely it is to be right), a high-stakes outcome or a request to appeal. Some schemes also review a random sample of routine cases to audit the rest. The reviewer approves, changes or rejects the output, and the record of each review shows how often people overrode the system. Review protects only where reviewers keep using independent judgement, so the design also has to guard against over-trust and fatigue.

Example Use Cases

Safety

Implementing mandatory human physician review for any medical AI diagnostic recommendation before treatment decisions are made, especially for complex cases or when the system confidence is below established thresholds, ensuring patient safety through expert oversight.

Transparency

Requiring human review of automated loan approval decisions when applicants request explanations or appeal rejections, allowing human underwriters to provide clear reasoning and ensure customers understand the decision-making process behind their application outcomes.

General

A recruitment agency requires a named recruiter to confirm or reverse every automated rejection before it is sent, and logs the decision and the reason. The agency's governance board reviews the override rate each quarter to see whether the screening tool and its reviewers are working as intended.

Limitations

  • Review capacity sets a ceiling. As volume rises, cases queue, decisions are delayed and time-sensitive uses become impractical, so the escalation rules must be tuned to what the available trained reviewers can handle.
  • Reviewers tire and drift. Attention and consistency fall over a long shift or a run of similar cases, so two reviewers, or one reviewer at different times, may decide the same case differently.
  • Automation bias. People tend to accept a system's recommendation, particularly under time pressure, so review can become a rubber stamp. A very high approval rate with almost no overrides may signal this rather than a good system.
  • The escalation rules decide what a human ever sees. A confidently wrong output passes without review when only low-confidence cases are escalated, so reviewing a random sample of routine cases is needed to find these errors.
  • Human review does not reliably improve decisions. A meta-analysis of 106 experiments found human-AI combinations performed worse on average than the better of human or AI alone, with losses in decision-making tasks. The effect of a checkpoint should be tested, not assumed.

Resources

Research Papers

A model for types and levels of human interaction with automation
R. Parasuraman, T.B. Sheridan, and C.D. Wickens•May 1, 2000

Technical developments in computer hardware and software now make it possible to introduce automation into virtually all aspects of human-machine systems. Given these technical capabilities, which system functions should be automated and to what extent? We outline a model for types and levels of automation that provides a framework and an objective basis for making such choices. Appropriate selection is important because automation does not merely supplant but changes human activity and can impose new coordination demands on the human operator. We propose that automation can be applied to four broad classes of functions: 1) information acquisition; 2) information analysis; 3) decision and action selection; and 4) action implementation. Within each of these types, automation can be applied across a continuum of levels from low to high, i.e., from fully manual to fully automatic. A particular system can involve automation of all four types at different levels. The human performance consequences of particular types and levels of automation constitute primary evaluative criteria for automation design using our model. Secondary evaluative criteria include automation reliability and the costs of decision/action consequences, among others. Examples of recommended types and levels of automation are provided to illustrate the application of the model to automation design.

On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
Sarah Sterz et al.•Jun 3, 2024

Tutorials

Human-in-the-loop (LangChain documentation)

Documentations

Article 14: Human Oversight | EU Artificial Intelligence Act

Tags

General Dimensions

Safety Dimensions

Transparency Dimensions