Runtime Monitoring and Circuit Breakers

Description

Runtime monitoring and circuit breakers observe a deployed AI service and restrict model calls when configured signals breach specified limits. In the closed state, requests pass through and recent failures are counted, while an open breaker rejects requests or provides a configured fallback, such as a cached answer, a simpler model, a rule-based default or referral to a person. After a waiting period, the half-open state permits a few trial calls and closes again only if they succeed. Model-related triggers can include immediately available signals such as input-drift scores (how far incoming data has moved from the training data) or low prediction confidence, since accuracy usually needs later outcome labels. Logs record state changes and triggering signals, supporting review of why the service was restricted without guaranteeing that the fallback is safe.

Example Use Cases

Safety

A hospital's diagnostic-support service stops showing model suggestions to clinicians when the share of low-confidence predictions or the input drift score passes preset limits, for example after a scanner software update changes the images. A notice tells staff the standard workflow applies until engineers reset the breaker.

Reliability

Deploying runtime monitoring for a recommendation engine that tracks recommendation diversity, click-through rates, and user engagement patterns, automatically switching to simpler algorithms when complex models show signs of performance degradation or unusual behaviour patterns.

Transparency

A bank's credit-scoring service records every breaker state change, the metric that triggered it and the fallback used in an audit log. Model risk reviewers read the log each month to see how often, when and why the model was restricted, and whether the thresholds need revisiting.

General

An airline's customer-service assistant is wrapped in a circuit breaker that counts timeouts, blocked outputs and refusals. When they pass a limit after a faulty release, traffic switches to a scripted menu and the on-call engineer is paged, so the faulty version is stopped quickly without waiting for a manual decision.

Limitations

Thresholds are hard to set. Too tight and the breaker trips on normal variation and withdraws a working service. Too loose and it stays closed through a real failure. Good settings need historical data from the live system, which is scarce at launch.
Model-level triggers are proxies for correctness. Drift scores and confidence can look normal while the model is confidently wrong, so a breaker that never trips is not evidence that predictions are right.
Gradual degradation and slow attacks can stay inside the limits, because each reading is acceptable while the cumulative effect is harmful. Catching them needs separate trend analysis or delayed accuracy checks alongside the breaker.
The breaker only moves the risk to the fallback. A cached answer, a simpler model or a human referral must itself be tested, and referral fails if reviewers cannot absorb a sudden diversion of traffic.
Tripping has a cost. A false trip withdraws a working service from users who may depend on it in time-critical settings, and reopening after a fixed cooldown, or on a few easy trial requests, can restore traffic before the underlying fault is fixed.

Resources

Research Papers

Release It! Design and Deploy Production-Ready Software

Michael Nygard•Jan 1, 2018

Design your application for maximum uptime, performance, and return on investment in the face of the harsh realities of the real world.

Hidden Technical Debt in Machine Learning Systems

D. Sculley et al.•Jan 1, 2015

Machine learning offers a fantastically powerful toolkit for building useful complexprediction systems quickly. This paper argues it is dangerous to think ofthese quick wins as coming for free. Using the software engineering frameworkof technical debt, we find it is common to incur massive ongoing maintenancecosts in real-world ML systems. We explore several ML-specific risk factors toaccount for in system design. These include boundary erosion, entanglement,hidden feedback loops, undeclared consumers, data dependencies, configurationissues, changes in the external world, and a variety of system-level anti-patterns.

Software Packages

pybreaker: Python implementation of the Circuit Breaker pattern

Oct 6, 2026

Python implementation of the Circuit Breaker pattern.

Evidently: open-source ML and LLM observability framework

Oct 9, 2026

Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

Tutorials

CircuitBreaker — Martin Fowler

Martin Fowler

You use software circuit breakers on connections to remote services. These breakers trip when the supplier becomes unresponsive, once tripped the breaker no longer calls the supplier until reset.

Documentations

Circuit Breaker pattern — Azure Architecture Center

claytonsiemens77

Learn how to handle faults that might take varying amounts of time to fix when applications connect to a remote service or resource.

Related Techniques

Name	Description	Assurance Goals
API Usage Pattern Monitoring	API usage pattern monitoring examines how each account uses a deployed model's API, to catch misuse that no single request reveals. The monitor keeps a running record per account and compares it with a baseline of normal use, raising an alert when the account departs from it. Published detectors for model theft show two ways to do this: one tests whether the distances between an account's successive queries follow the bell-shaped pattern that ordinary use produces, and another estimates how much of the model's behaviour the account's queries and responses would let someone rebuild. Providers also score how often an account triggers content filters or repeats jailbreak attempts, and send high scores for review. The output is a log of alerts and the actions taken, which gives a dated record of how the API was used compared with the uses it was approved for.	Security Safety Transparency
Signal Temporal Logic Monitoring	Signal temporal logic (STL) monitoring checks the signals a system produces over time, such as speeds, distances, positions or a model's outputs, against safety requirements written in signal temporal logic, a formal language for stating how real-valued signals must behave over time intervals (for example, 'whenever the gap to the vehicle ahead falls below 10 m, speed drops below 5 m/s within 2 s'). Each requirement is built from comparisons of signals with thresholds, joined by logical operators and by timed operators such as 'always within [a, b]', 'eventually within [a, b]' and 'until'. For each trace the monitor computes a robustness value, a margin to the requirement rather than robustness to adversarial inputs: its sign says whether the requirement is met (positive) or violated (negative), and its size is a tolerance: any signal whose values stay closer than that to the observed ones at every instant gets the same verdict. Monitors run offline on logged or simulated traces, or online on live signals, where a requirement that looks ahead in time can be decided only once enough of the signal has arrived. RTAMT, the named tool, is an open-source Python library with an optional C++ back-end that builds such monitors from a written specification and connects to ROS and MATLAB/Simulink; its online monitors accept only requirements whose future time windows are bounded and give each verdict after a delay equal to the requirement's time horizon, and it reports the robustness value but does not act on it, so stopping or correcting the system is left to a separate safeguard.	Safety
Out-of-Domain Detection	Out-of-domain detection checks whether a user's query falls inside the topics and tasks a conversational AI system is meant to handle. Scope is defined by examples, such as an assistant's supported intents (the actions it can recognise), and a detector then scores how closely each new query resembles them. Typical detectors set a minimum confidence on an intent classifier, add an explicit out-of-scope class trained on examples of unsupported queries, or measure how far a query's embedding, a numerical summary of its meaning, lies from the in-scope examples. A query judged out of scope can be declined, handed to a person or another service, or answered with a request to rephrase, instead of receiving an unreliable or invented answer. The detector tests resemblance to the defined scope, not what the system actually knows, so it needs evaluating on realistic out-of-scope queries.	Reliability Safety Transparency