Human-in-the-Loop Safeguards
Description
Human-in-the-loop safeguards hold an AI system's output at a checkpoint until a trained person has reviewed it. Designers set the rules that send a case to review, such as low model confidence (the system's own estimate of how likely it is to be right), a high-stakes outcome or a request to appeal. Some schemes also review a random sample of routine cases to audit the rest. The reviewer approves, changes or rejects the output, and the record of each review shows how often people overrode the system. Review protects only where reviewers keep using independent judgement, so the design also has to guard against over-trust and fatigue.
Example Use Cases
Safety
Implementing mandatory human physician review for any medical AI diagnostic recommendation before treatment decisions are made, especially for complex cases or when the system confidence is below established thresholds, ensuring patient safety through expert oversight.
Transparency
Requiring human review of automated loan approval decisions when applicants request explanations or appeal rejections, allowing human underwriters to provide clear reasoning and ensure customers understand the decision-making process behind their application outcomes.
General
A recruitment agency requires a named recruiter to confirm or reverse every automated rejection before it is sent, and logs the decision and the reason. The agency's governance board reviews the override rate each quarter to see whether the screening tool and its reviewers are working as intended.
Limitations
- Review capacity sets a ceiling. As volume rises, cases queue, decisions are delayed and time-sensitive uses become impractical, so the escalation rules must be tuned to what the available trained reviewers can handle.
- Reviewers tire and drift. Attention and consistency fall over a long shift or a run of similar cases, so two reviewers, or one reviewer at different times, may decide the same case differently.
- Automation bias. People tend to accept a system's recommendation, particularly under time pressure, so review can become a rubber stamp. A very high approval rate with almost no overrides may signal this rather than a good system.
- The escalation rules decide what a human ever sees. A confidently wrong output passes without review when only low-confidence cases are escalated, so reviewing a random sample of routine cases is needed to find these errors.
- Human review does not reliably improve decisions. A meta-analysis of 106 experiments found human-AI combinations performed worse on average than the better of human or AI alone, with losses in decision-making tasks. The effect of a checkpoint should be tested, not assumed.
Resources
Research Papers
A model for types and levels of human interaction with automation
Technical developments in computer hardware and software now make it possible to introduce automation into virtually all aspects of human-machine systems. Given these technical capabilities, which system functions should be automated and to what extent? We outline a model for types and levels of automation that provides a framework and an objective basis for making such choices. Appropriate selection is important because automation does not merely supplant but changes human activity and can impose new coordination demands on the human operator. We propose that automation can be applied to four broad classes of functions: 1) information acquisition; 2) information analysis; 3) decision and action selection; and 4) action implementation. Within each of these types, automation can be applied across a continuum of levels from low to high, i.e., from fully manual to fully automatic. A particular system can involve automation of all four types at different levels. The human performance consequences of particular types and levels of automation constitute primary evaluative criteria for automation design using our model. Secondary evaluative criteria include automation reliability and the costs of decision/action consequences, among others. Examples of recommended types and levels of automation are provided to illustrate the application of the model to automation design.
Human-in-the-loop machine learning: a state of the art
Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them.