Safety Envelope Testing

Description

Safety envelope testing checks how an AI system performs at and beyond the edges of the conditions it was designed for, so that failure modes are found before deployment. Testers first write down those conditions, called the operational design domain, such as the road types, weather, lighting and speeds a vehicle is built for. They then build scenarios that push each condition towards its limit and past it, and measure how performance degrades as conditions get harder, repeating runs where the system does not behave the same way twice. The results show where, within the conditions tested, the system moves from acceptable to unsafe behaviour. They support operating limits, safety margins and a point at which the system should hand over to a person or stop.

Example Use Cases

Safety

Testing autonomous vehicle perception systems at the limits of weather conditions, lighting, and sensor coverage to establish safe operational boundaries and determine when human intervention is required.

A trading firm replays historical crash periods and synthetic shocks with volatility pushed beyond the range its trading model was developed on, to check that position limits and the automatic trading halt trigger before losses reach the agreed ceiling.

Reliability

A radiology team tests a chest X-ray triage model on scans at the edge of its intended use, such as the poorest image quality and the oldest scanner models it was approved for, and records where sensitivity falls below the agreed floor, so that scans beyond that point are referred to a radiologist.

Limitations

Results are only as good as the operational design domain used. If the specification omits a condition the system meets in service, such as a road type or a market regime, the tests probe the wrong edge and a clean result gives false confidence.
Coverage is incomplete by construction. Conditions interact across many dimensions, so the number of boundary combinations grows quickly, and failures that appear only when several factors reach their limits together can be missed.
Performance of a learned system need not fall smoothly towards the boundary. Failures can sit inside the envelope, so good results at the edge do not show that the interior is safe.
Results describe one system version in one test setting, and a single pass proves little for a system that behaves differently on successive runs, which needs repeated trials. Retraining, new sensors or a gap between simulation and the real world can move the boundary, so the tests need repeating after each material change.
The technique shows where the system stops performing acceptably. It does not keep the system inside that envelope in service, which needs a separate runtime monitor or out-of-distribution detector.

Resources

Research Papers

Challenges in Autonomous Vehicle Testing and Validation

Philip Koopman and Michael Wagner•Apr 5, 2016

Software testing is all too often simply a bug hunt rather than a well-considered exercise in ensuring quality. A more methodical approach than a simple cycle of system-level test-fail-patch-test will be required to deploy safe autonomous vehicles at scale. The ISO 26262 development V process sets up a framework that ties each type of testing to a corresponding design or requirement document, but presents challenges when adapted to deal with the sorts of novel testing problems that face autonomous vehicles. This paper identifies five major challenge areas in testing according to the V model for autonomous vehicles: driver out of the loop, complex requirements, non-deterministic algorithms, inductive learning algorithms, and fail-operational systems. General solution approaches that seem promising across these different challenge areas include: phased deployment using successively relaxed operational scenarios, use of a monitor/actuator pair architecture to separate the most complex autonomy functions from simpler safety functions, and fault injection as a way to perform more efficient edge case testing. While significant challenges remain in safety-certifying the type of algorithms that provide high-level autonomy themselves, it seems within reach to instead architect the system and its accompanying design process to be able to employ existing software safety approaches.

Documentations

BS ISO 34503:2023 Road vehicles. Test scenarios for automated driving systems. Specification for operational design domain

Related Techniques

Name	Description	Assurance Goals
Scenario-Based Simulation Testing	Scenario-based simulation testing checks an automated system, most often an automated driving system, by running it through a planned set of situations in a simulator and judging each run against criteria fixed in advance. Scenarios are written at three levels of detail: a functional scenario describes a situation in words (a car cuts in ahead on a motorway), a logical scenario turns it into parameters with ranges (speeds, gaps, road friction), and a concrete scenario fixes one value for every parameter so that the run can be reproduced exactly. Functional scenarios are derived from hazard analysis and from the operational design domain (the conditions the system is designed to work in), and concrete scenarios are sampled from the logical ranges, either to cover the whole range or to seek out rare, critical combinations, or are reconstructed from recorded incidents. Each concrete scenario is executed with the system under test in the loop, and pass or fail is decided against its expected behaviour, such as no collision, or by criticality measures such as time-to-collision, giving a record of which situations the system handles and which it does not. ISO 34502 gives a scenario-based safety evaluation framework of this kind for automated driving systems on limited-access highways.	Safety
Multi-Agent System Testing	Multi-agent system testing examines systems in which several AI agents act on shared tasks or resources, to find failures that no single agent shows on its own. Testers run the agents together in simulated or scripted scenarios, often with injected faults such as failed sensors, delayed messages or conflicting goals, and record the failures that appear. Checks cover whether the task is completed, whether coordination breaks down (for example through deadlock, collisions or one agent's error spreading to others), and whether stated safety rules are ever broken. For language-model agents, benchmarks such as MultiAgentBench score task completion and the quality of collaboration or competition. For small or abstract models, a model checker such as MCMAS can search every reachable state for a violation of a rule written in formal logic and, for many kinds of rule, return an example run that breaks it.	Safety Reliability
Metamorphic Testing	Metamorphic testing checks a model or AI system without needing the correct output for each test input, by checking that the outputs for related inputs stand in a relation that must hold if the system works as intended. The tester first writes metamorphic relations from knowledge of the task, each pairing a change to the input with the change it should cause in the output, or with no change: adding fog to a road image should leave a steering prediction within a set tolerance, adding a few stray sensor points outside the road area should not remove any detected obstacle, and rotating a whole flight scenario should leave its outcome unchanged. Existing test inputs (source cases) are transformed into follow-up inputs according to each relation, the system is run on both, and each pair is checked, with numeric outputs often allowed a tolerance. A violation points to a fault or unwanted behaviour without any labelled answer, unless the relation itself is wrong; the output is the violation rate for each relation with the failing pairs, and a relation that holds on every pair shows only that the outputs agree with each other, not that they are right. Prompt robustness testing and causal discrimination testing in this catalogue each apply one family of invariance relations (meaning-preserving changes to a prompt, and a change of protected attribute), and prompt robustness testing also scores the answers against known ones; metamorphic testing is the general procedure for writing and checking any such relation, including relations that say how the output should change, and needs no known answers.	Reliability