Scenario-Based Simulation Testing

Description

Scenario-based simulation testing checks an automated system, most often an automated driving system, by running it through a planned set of situations in a simulator and judging each run against criteria fixed in advance. Scenarios are written at three levels of detail: a functional scenario describes a situation in words (a car cuts in ahead on a motorway), a logical scenario turns it into parameters with ranges (speeds, gaps, road friction), and a concrete scenario fixes one value for every parameter so that the run can be reproduced exactly. Functional scenarios are derived from hazard analysis and from the operational design domain (the conditions the system is designed to work in), and concrete scenarios are sampled from the logical ranges, either to cover the whole range or to seek out rare, critical combinations, or are reconstructed from recorded incidents. Each concrete scenario is executed with the system under test in the loop, and pass or fail is decided against its expected behaviour, such as no collision, or by criticality measures such as time-to-collision, giving a record of which situations the system handles and which it does not. ISO 34502 gives a scenario-based safety evaluation framework of this kind for automated driving systems on limited-access highways.

Example Use Cases

Safety

A developer of a motorway driving function derives cut-in, cut-out and stopped-vehicle scenarios from its hazard analysis, gives each logical scenario ranges for speeds, gaps and road friction, and runs several thousand sampled concrete scenarios in simulation for every software release, holding back any release in which a run ends in a collision the function was required to avoid.

A robotaxi operator rebuilds the fatal crashes recorded over a decade in the city where it operates as concrete simulation scenarios, and replays each one with its driving system in place of first one human driver and then the other, so that it can show regulators how the system behaves in the crash types that actually occur in its operating area.

An operator of an autonomous passenger ferry tests its collision-avoidance planner on head-on, crossing and overtaking encounters described in the international collision regulations, varying the other vessel's speed, bearing and visibility in a ship simulator, and checks in every run that the ferry keeps its required passing distance and makes the manoeuvre the regulations call for.

Limitations

The evidence covers only the scenarios that were specified. Real operating conditions cannot be listed exhaustively, so a clean result shows that the system handles the catalogue, not that it is safe; the safety argument depends on how completely the functional scenarios were derived from the hazard analysis and operational design domain, which needs independent review.
Results are only as credible as the simulation. If the models of sensors, vehicle dynamics or other road users have not been validated against the real world, a system can pass in simulation and fail in operation, so key scenarios should be repeated on a test track or against recorded real data.
How concrete scenarios are sampled shapes what the results mean. Covering whole parameter ranges ignores how often each situation occurs, and searching for critical corner cases finds failures efficiently but over-represents rare situations, so neither supports a statement about real-world accident rates unless results are weighted by how often scenarios occur.
Pass criteria encode a judgement about acceptable behaviour. 'No collision' misses near misses and lucky escapes, while criticality measures such as time-to-collision need thresholds that the developer must choose and justify, and the verdicts change with them.
Building scenario catalogues, simulation environments and behaviour models for other road users takes specialist effort and substantial computing. ISO 34502 applies only to limited-access highways, so for urban roads, ships and other domains the scenario structure, pass criteria and coverage argument have to be built without the standard's support.

Resources

Research Papers

Scenarios for Development, Test and Validation of Automated Vehicles

Till Menzel, Gerrit Bagschik, and Markus Maurer•Jun 1, 2018

The latest version of the ISO 26262 standard from 2016 represents the state of the art for a safety-guided development of safety-critical electric/electronic vehicle systems. These vehicle systems include advanced driver assistance systems and vehicle guidance systems. The development process proposed in the ISO 26262 standard is based upon multiple V-models, and defines activities and work products for each process step. In many of these process steps, scenario based approaches can be applied to achieve the defined work products for the development of automated driving functions. To accomplish the work products of different process steps, scenarios have to focus on various aspects like a human understandable notation or a description via state variables. This leads to contradictory requirements regarding the level of detail and way of notation for the representation of scenarios. In this paper, the authors discuss requirements for the representation of scenarios in different process steps defined by the ISO 26262 standard, propose a consistent terminology based on prior publications for the identified levels of abstraction, and demonstrate how scenarios can be systematically evolved along the phases of the development process outlined in the ISO 26262 standard.

Software Packages

carla-simulator/scenario_runner: Traffic scenario definition and execution engine

Sep 28, 2026

Traffic scenario definition and execution engine

Tutorials

Create a new scenario tutorial — ScenarioRunner

Documentations

Road vehicles — Test scenarios for automated driving systems — Scenario based safety evaluation framework

Jan 1, 2022

Related Techniques

Name	Description	Assurance Goals
Safety Envelope Testing	Safety envelope testing checks how an AI system performs at and beyond the edges of the conditions it was designed for, so that failure modes are found before deployment. Testers first write down those conditions, called the operational design domain, such as the road types, weather, lighting and speeds a vehicle is built for. They then build scenarios that push each condition towards its limit and past it, and measure how performance degrades as conditions get harder, repeating runs where the system does not behave the same way twice. The results show where, within the conditions tested, the system moves from acceptable to unsafe behaviour. They support operating limits, safety margins and a point at which the system should hand over to a person or stop.	Safety Reliability
Dangerous Capability Evaluation	Dangerous capability evaluation measures whether a frontier AI model can carry out tasks that could enable severe harm, such as finding and exploiting software vulnerabilities, persuading or deceiving people, acquiring resources and running copies of itself, or helping to plan an attack with chemical, biological, radiological or nuclear materials, and compares the results with capability thresholds that the developer's safety framework fixes before testing. Evaluators first write a threat model for each risk domain; the framework names a critical capability level, at which the model could pose an extreme risk without extra safeguards, and an early-warning trigger defined on the evaluation results, which the method's originators sketched for one domain as success on more than half of a set of tasks given ten attempts at each. They then build task suites for each domain, such as capture-the-flag hacking challenges in sandboxed networks or multi-step tasks that require the model to set up a cloud server and install software, and try to draw out the model's best performance: it is tested without the safety filters of the deployed product, given agent scaffolding (prompts, planning loops and tools such as a command shell) and sometimes fine-tuned, because the evaluation must not underestimate what the model can do. Scoring records success over repeated attempts and, for tasks the model cannot finish, how close it came, by starting it from intermediate milestones and measuring how much expert help it needs to succeed. Unlike red teaming, which probes a system and its safeguards for whatever failures can be found, and unlike tests of whether a model refuses harmful requests, it measures what the model is able to do, and whether the results cross the trigger informs decisions on deployment and on the security measures the model needs.	Safety
Multi-Agent System Testing	Multi-agent system testing examines systems in which several AI agents act on shared tasks or resources, to find failures that no single agent shows on its own. Testers run the agents together in simulated or scripted scenarios, often with injected faults such as failed sensors, delayed messages or conflicting goals, and record the failures that appear. Checks cover whether the task is completed, whether coordination breaks down (for example through deadlock, collisions or one agent's error spreading to others), and whether stated safety rules are ever broken. For language-model agents, benchmarks such as MultiAgentBench score task completion and the quality of collaboration or competition. For small or abstract models, a model checker such as MCMAS can search every reachable state for a violation of a rule written in formal logic and, for many kinds of rule, return an example run that breaks it.	Safety Reliability