Metamorphic Testing

Description

Metamorphic testing checks a model or AI system without needing the correct output for each test input, by checking that the outputs for related inputs stand in a relation that must hold if the system works as intended. The tester first writes metamorphic relations from knowledge of the task, each pairing a change to the input with the change it should cause in the output, or with no change: adding fog to a road image should leave a steering prediction within a set tolerance, adding a few stray sensor points outside the road area should not remove any detected obstacle, and rotating a whole flight scenario should leave its outcome unchanged. Existing test inputs (source cases) are transformed into follow-up inputs according to each relation, the system is run on both, and each pair is checked, with numeric outputs often allowed a tolerance. A violation points to a fault or unwanted behaviour without any labelled answer, unless the relation itself is wrong; the output is the violation rate for each relation with the failing pairs, and a relation that holds on every pair shows only that the outputs agree with each other, not that they are right. Prompt robustness testing and causal discrimination testing in this catalogue each apply one family of invariance relations (meaning-preserving changes to a prompt, and a change of protected attribute), and prompt robustness testing also scores the answers against known ones; metamorphic testing is the general procedure for writing and checking any such relation, including relations that say how the output should change, and needs no known answers.

Example Use Cases

Reliability

An air navigation service provider evaluating a machine-learning conflict-detection aid for en-route controllers has no ground truth for most replayed traffic samples. Its test team writes three relations: swapping two aircraft's call signs, shifting every timestamp by the same amount, and rotating the whole sample together with its wind field must not change which aircraft pairs are flagged. It runs them on a year of recorded traffic and holds the release until every violation is explained.

A supplier of lidar obstacle perception for driverless shuttles cannot label every frame it records. Following a relation implied by the perception specification, its test team adds a few hundred random points outside the mapped road area to recorded frames and checks that no obstacle detected in the original frame disappears, reporting the share of frames where one does, by number of points added, as a release criterion for each software build.

A weather service testing a neural network that nowcasts rainfall from radar images checks a relation expected from the way rain moves with the wind: away from the domain's edges, shifting the input radar sequence a few grid cells east should shift the forecast rain field by the same amount, within a tolerance set from the model's day-to-day error. Forecasters review the breaking cases to judge whether the model has learnt a dependence on location, such as on terrain, that it should keep.

A bank moving its credit-scoring models to a new library cannot treat every difference from the old library's outputs as a fault, since the solvers differ. Its validation team writes relations the mathematics guarantees: reordering the input columns, or adding a constant to a numeric input in training and scoring data of a logistic regression with an unpenalised intercept, must leave every predicted probability unchanged within solver tolerance. Each violation is investigated as a fault.

Limitations

Passing shows that outputs are consistent, not that they are correct. A model that gives the same wrong answer for a source case and its follow-up satisfies the relation, and one that is wrong in the same way everywhere can pass every invariance relation. On its own the technique therefore does not suit systems where each output must be shown to be right, and it complements rather than replaces testing against known answers where those exist.
The evidence is only as good as the relations, which are written by hand from domain knowledge, with no systematic way to choose among the many possible ones. A relation the system is not bound to obey produces false alarms: in a study of the Weka library, violations by its nearest-neighbour classifier came from legitimate tie-breaking between equally likely labels, not from faults. An incomplete set of relations gives false confidence.
Most model outputs need a tolerance, and the result depends heavily on it. In the DeepTest study of three driving models, violations under simple image transformations fell from 23,391 when a transformed frame's squared error against the human steering label could be at most the original mean squared error to 330 when it could be five times that, and manual review still found false positives. Counts compare only between runs with the same tolerance, set from what the application can accept.
Follow-up inputs are synthetic and may not occur in practice: rain or fog added to an image, or points scattered into a lidar frame, approximate real conditions without reproducing them. A violation on an unrealistic input may not matter in operation, and a pass covers only the transformations tried, so the evidence is about the tested relations and transformations, not the operating conditions as a whole.
A model whose outputs vary between runs on the same input, through sampling, seeds or GPU arithmetic, breaks relations by chance, so runs must be made repeatable or each case repeated and judged statistically. This bites hardest for relations over the training data, such as Xie et al.'s for classifiers, which need the model retrained for every follow-up case; for models that are costly to train only prediction-time relations are practical, and they test the trained model, not its training.

Resources

Research Papers

Metamorphic Testing: A New Approach for Generating Next Test Cases

T. Y. Chen, S. C. Cheung, and S. M. Yiu•Feb 28, 2020

In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.

Testing and validating machine learning classifiers by metamorphic testing

Xiaoyuan Xie et al.•Apr 1, 2011

Tutorials

Metamorphic Testing: Testing the Untestable

Sergio Segura et al.•May 1, 2020

Documentations

ISTQB Certified Tester AI Testing (CT-AI) Syllabus v2.0, section 6.1.5 Metamorphic Testing

International Software Testing Qualifications Board (ISTQB)•Apr 17, 2026

The CT-AI v1.0 certification will be retired: the English version will remain available until April 21, 2027, and non-English versions will remain available

Related Techniques

Name	Description	Assurance Goals
Safety Envelope Testing	Safety envelope testing checks how an AI system performs at and beyond the edges of the conditions it was designed for, so that failure modes are found before deployment. Testers first write down those conditions, called the operational design domain, such as the road types, weather, lighting and speeds a vehicle is built for. They then build scenarios that push each condition towards its limit and past it, and measure how performance degrades as conditions get harder, repeating runs where the system does not behave the same way twice. The results show where, within the conditions tested, the system moves from acceptable to unsafe behaviour. They support operating limits, safety margins and a point at which the system should hand over to a person or stop.	Safety Reliability
Prompt Robustness Testing	Prompt robustness testing measures how much a language model's answers, or its performance on a task, change when the prompt is varied in ways that should not matter, such as rewording, formatting, capitalisation, reordering, or the typos and synonym swaps real users make. The tester starts from task prompts with known correct answers, writes variants that keep the meaning from templates or generates them with perturbation tools such as PromptBench, and runs the model on every variant. Some tools search for the variant that does the most damage, so a drop under that kind of attack is a worst case, not a typical failure rate. The result is reported as the spread of performance across variants, for example the gap between the best and worst format, together with the variants that changed the answer. On one widely used open model, accuracy differed by up to 76 points across formatting changes that kept the meaning, which is why one prompt template is not enough.	Reliability
Multi-Agent System Testing	Multi-agent system testing examines systems in which several AI agents act on shared tasks or resources, to find failures that no single agent shows on its own. Testers run the agents together in simulated or scripted scenarios, often with injected faults such as failed sensors, delayed messages or conflicting goals, and record the failures that appear. Checks cover whether the task is completed, whether coordination breaks down (for example through deadlock, collisions or one agent's error spreading to others), and whether stated safety rules are ever broken. For language-model agents, benchmarks such as MultiAgentBench score task completion and the quality of collaboration or competition. For small or abstract models, a model checker such as MCMAS can search every reachable state for a violation of a rule written in formal logic and, for many kinds of rule, return an example run that breaks it.	Safety Reliability