System-Theoretic Process Analysis

Description

System-Theoretic Process Analysis (STPA) is a hazard analysis method developed by Nancy Leveson at MIT that treats safety as a control problem: a loss can occur when parts of a system that have not failed interact unsafely, for example when a controller acts on a wrong belief about the process it controls. The analysis has four steps: define the losses to prevent and the system-level hazards that lead to them; model the system as a control structure of controllers, the control actions they issue and the feedback they receive; identify unsafe control actions; and write loss scenarios explaining how each could arise. An unsafe control action is one of four types (not providing an action causes a hazard, providing it causes a hazard, providing it too early, too late or in the wrong order, or stopping it too soon or applying it too long), and loss scenarios cover causes such as missing or wrong feedback, a flawed control algorithm, or a controller's process model (its internal picture of what it controls) that no longer matches reality. In a system with machine-learning components, a learned model appears in the control structure as a controller whose control algorithm and process model were learned from data, or as a feedback source, such as a perception model, whose errors mislead a controller; loss scenarios include training data that missed an operating condition, a shift from training to deployment conditions, and operators who over-rely on the model, and published adaptations also model data collection, training and evaluation as control loops, so that a scenario can trace an unsafe control action back to a development decision such as an incomplete reward function. The output is a traceable set of unsafe control actions, loss scenarios and the safety constraints and requirements derived from them, which the team uses to change the design and to decide what to test; it analyses a model of the system before or alongside testing, and it gives no probabilities.

Example Use Cases

Safety

An automotive supplier building a pedestrian collision-avoidance function around a neural-network detector models the vehicle's control structure with the detector as the braking controller's source of feedback. It finds the unsafe control action 'emergency braking not applied while a pedestrian is crossing', with scenarios such as missed detections of partly hidden pedestrians at dusk, and turns them into requirements on training-data coverage and on the detector's miss rate.

A hospital's patient-safety team preparing to deploy a machine-learning early-warning score for patient deterioration models the score, the nurses who receive its alerts and the doctors who act on them as one control structure. It finds unsafe control actions such as an escalation that comes too late because vital signs reach the record in batches, and alerts ignored after repeated false alarms, and changes data entry and the alerting rules before go-live.

A frontier AI developer's safety team analysing internal use of an AI coding agent models the engineers who prompt it, the automated monitor that scores its sessions, the human reviewers and the merge gate as controllers of the company codebase. It finds unsafe control actions such as a monitoring alert that arrives after the agent's change has been merged, and sets requirements on how quickly sessions are reviewed and on reviewers' independence from the agent.

An engineering team training a reinforcement-learning policy to land a drone splits the mission into approach, descent and touchdown, identifies unsafe control actions for each phase, such as descending too fast close to the landing pad, and uses the loss scenarios to choose the wind and sensor-noise perturbations for its simulation tests and to adjust the reward before the next round of training.

Limitations

The analysis depends on judgement at every step: deciding which losses matter, where the system ends, which parts act as controllers and which scenarios to record. Separate teams analysing the same system can reach different models and scenario lists, so the work needs review by people outside the team, and a short list of loss scenarios is not evidence that no others exist.
STPA does not estimate how likely a hazard or scenario is, and its authors argue that such estimates are unreliable for software-intensive systems. Where an assurance case or regulator needs a quantitative target, such as a failure rate per hour, it must be paired with a quantitative method, and STPA alone cannot show that a scenario is too rare to need a mitigation.
For a learned component the analysts cannot read the process model or the control algorithm, because both are encoded in trained weights. Scenarios explaining why the model would act unsafely are therefore hypotheses about its training data and operating conditions until tests or monitoring confirm them, and for policies with continuous states and actions the analysis has to be split by mission phase or subtask to stay manageable.
A control structure that covers only the system in operation misses causes that lie in how the model was built, such as data collection, labelling, reward design or the choice of evaluation set; a survey of 31 studies applying STPA to learning-enabled systems found that most modelled operation only. Adding the development lifecycle as control loops finds these causes but makes the analysis substantially larger.
A full analysis of a complex system takes sustained effort from people trained in the method, working with domain and machine-learning experts. A language model can draft loss scenarios faster, but its drafts steer the team towards the scenarios it happens to suggest and can leave gaps, so they need the same review as scenarios written by hand.

Resources

Research Papers

STPA: A New Hazard Analysis Technique

Nancy G. Leveson•Jan 13, 2012

Software Packages

PASTA: Pragmatic-Automated System Theoretic Process Analysis (VS Code extension)

VS Code extension for STPA-based risk analyses with interactive diagrams and integrations for fault trees, safe-by-construction behavior model generation, and verification. - kieler/pasta

Tutorials

Introduction to STPA: Anticipating and Preventing Loss Scenarios in Complex Systems (MIT STAMP Workshop tutorial)

Documentations

STPA Handbook

Related Techniques

Name	Description	Assurance Goals
Fault Tree Analysis	Fault tree analysis (FTA) is a deductive, top-down method that starts from one undesired system-level event, such as an automated braking function failing to brake for an obstacle, and works backwards to the combinations of lower-level events that could cause it. The analyst builds a tree of logic gates beneath that top event: an OR gate means any one of its inputs is enough to cause the event above it, an AND gate means all of its inputs must occur together, and the branches end in basic events such as a hardware failure, a human error or a software error. Solving the tree gives its minimal cut sets, the smallest combinations of basic events that produce the top event (a set with a single event is a single point of failure), and, when each basic event has a probability, the probability of the top event, which can be checked against a target or used to set a probability budget for each part. In a system with a machine-learning component, the model's errors enter the tree as basic events of their own, which unlike random hardware failures happen during normal operation at a rate that has to be measured on test or operational data, often depend on conditions such as weather or occlusion that can be added beneath it as conditioning events, and can be correlated from one input to the next. Working down from an acceptable top-event probability turns the model's share into a budget for its error rate, which becomes a performance requirement on the model; the tree shows where model errors matter in the system and how much error it can tolerate, but not why the model makes them.	Safety
Machine Learning Failure Mode and Effects Analysis	Machine Learning Failure Mode and Effects Analysis (ML FMEA) applies the process form of failure mode and effects analysis, a long-established worksheet method in which a team lists what could go wrong at each step of a process, rates each failure and acts on the worst first, to the pipeline that produces a machine-learning model rather than to the model alone. The team works through the pipeline one step at a time (in the 2025 reference version: collect data requests, collect data, ingest data, validate data, preprocess data, train, tune, analyse, deploy and validate the model, and analyse feedback from the field) and, prompted by guide words such as missing, incorrect or too late, records for each step how the resulting model could fail, the effect on the wider system, the pipeline causes, and the current practices that would prevent or catch the failure. Each row is rated for severity, occurrence and detection, and the product of the three, the risk priority number, ranks the rows so that recommended actions and owners go to the highest risks first; severity is taken from the system-level hazard analysis, because the harm a model failure can do depends on the system it sits in. The method's authors publish a pre-populated template listing typical failure modes, causes and mitigations for each step, which a team tailors to its own model and application. The result is a worksheet of ranked pipeline failure modes, their controls and assigned actions, kept up to date as the pipeline changes; it looks at failures one at a time in how the model is built and does not by itself analyse how the deployed system behaves.	Safety
STRIDE-AI Threat Modelling	STRIDE-AI is a threat-modelling method for machine-learning (ML) systems, proposed by Mauri and Damiani (2021, extended 2022), that lists, before any testing, the ways an attacker could compromise each asset of an ML pipeline and how serious each would be. It adapts Microsoft's STRIDE scheme, in which each of six threat types (spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege) breaks one security property (authenticity, integrity, non-repudiation, confidentiality, availability and authorisation), and gives each property an ML-specific definition: confidentiality, for example, means that querying a model reveals nothing beyond its inputs and outputs. The analyst surveys the assets produced across the ML life cycle, such as raw, labelled and validation data, hyperparameters, trained and deployed models and the model architecture, and with each asset's owners runs a failure mode and effects analysis (FMEA, a structured review of how a component could fail and with what effect), answering set questions about how the asset could fail, be misused or be abused. Each failure mode is mapped to the properties it would break and so to STRIDE threats, using the method's published tables of asset types, properties and known attacks; the analyst then works with developers to draw threat trees showing the conditions under which each threat could be realised, and rates each threat's priority (the worked example uses DREAD, which scores five factors from 1 to 10). The output is a threat model, a table of assets, threatened properties, threats and priorities with their threat trees: the method attacks and measures nothing, and choosing and testing controls for the threats it finds is left to other techniques.	Security