Red Teaming

Description

Red teaming is a structured exercise in which a team deliberately tries to make an AI system fail, so that weaknesses are found before real users or attackers find them. The organisers set the scope and the harms of concern, recruit people with a mix of skills (for example security testers, domain specialists and members of affected groups), and give them a way to log each attempt and judge whether it worked. Participants try manipulative prompts, awkward edge cases and requests designed to draw out harmful, insecure or discriminatory responses, combining many approaches where a test for one known weakness, such as prompt injection, covers only one. The result is a record of the failures found and how easily each was provoked, which is not a pass or fail verdict. It feeds a risk register (the list of known risks and who owns them) and fixes, and the exercise is repeated when the system changes.

Example Use Cases

Safety

A hospital convenes clinicians, medical students and engineers to stress-test a language model that drafts clinical notes, using real patient cases. Each response that could harm a patient, such as a wrong dosage, is logged and sent to the clinical safety officer to decide on fixes and on how staff may use the tool.

Security

Before launch, a bank's security team and outside testers spend two weeks trying to make its customer-service assistant leak client data or obey instructions hidden in uploaded documents. Each successful attack is written up with the steps to reproduce it and ranked by how easily it was exploited, so engineers can fix the worst first.

Fairness

A local authority asks advice charities and its own caseworkers to probe a chatbot that explains benefit entitlements. They look for answers that differ or discourage claims depending on a person's nationality, disability or first language, and record each case with the prompt that provoked it, for the authority to fix or justify.

General

An insurer's AI governance committee requires a red-team report before any customer-facing model goes live. Findings are entered in the risk register with an owner and a due date, and the committee signs off only when each high-severity item is fixed or formally accepted.

Limitations

Needs people who understand both the system and how attackers work, and some harms need domain specialists to judge, for example whether chemical instructions are accurate. Exercises are therefore costly and hard to scale across many systems.
Finds only what the team thinks to try. In Anthropic's original exercise with 324 crowd workers, no attacks targeted code generation although the models had been trained partly on code, so coverage depends on who is recruited and how they are briefed.
Results describe one model version, configuration and set of safeguards at one moment. After an update or a prompt change, fixed failures can return and new ones appear, so the exercise has to be repeated, which adds cost each time.
Tester-written attacks may differ from real misuse, and there is no agreed measure of success. In Anthropic's study, participants' own ratings and an automated harm score correlated only imperfectly, and a minority of participants produced most attacks, so exercises are hard to compare.
A report with few findings can be read as proof of safety when it shows only that this team did not find a failure in the time available. Reports should state scope, effort and what was not tried, or they invite false confidence.

Resources

Research Papers

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Deep Ganguli et al.•Nov 22, 2022

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

Software Packages

PyRIT: The Python Risk Identification Tool for generative AI

Feb 1, 2024

The Python Risk Identification Tool for generative AI (PyRIT) is an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems.

Tutorials

Red Teaming LLM Applications - DeepLearning.AI

Matteo Dora and Luca Martial•Jan 1, 2025

Documentations

OWASP GenAI Red Teaming Guide

OWASPGenAIProject Editor

Discover the GenAI Red Teaming Guide for comprehensive strategies to identify and mitigate security risks in AI-driven systems through red teaming techniques.

Related Techniques

Name	Description	Assurance Goals
Jailbreak Resistance Testing	Jailbreak resistance testing checks whether a language model or chatbot can be talked out of its safety rules. Testers write or collect prompts that try to elicit content the system is meant to refuse, such as role-playing a character without restrictions, framing the request as hypothetical, hiding it in an encoding such as Base64, or combining several of these. Each response is judged as a refusal, a breach or unclear, by a reviewer or an automated check. The result is the rate at which each attack family succeeds, with the example prompts that worked. This shows where the safeguards fail, but only for the attacks tried.	Safety Security
Adversarial Training Evaluation	Adversarial training evaluation tests whether a model trained on adversarial examples (inputs altered slightly to cause errors) is really more robust, or has only learned to resist the attacks used in training. The evaluator first states a threat model, meaning the kind and size of change allowed and what the attacker knows. The model is then attacked with a varied set of strong methods, including ones not used in training, ones adapted to this particular defence, and ones that do not rely on the model's gradients, which exposes models that merely hide them. The main output is robust accuracy, the share of inputs still classified correctly under attack, plotted against the allowed size of the change and reported next to accuracy on unaltered inputs. Because attacks can only find weaknesses, the result is an upper bound on robustness, not a guarantee.	Security
Model Extraction Defence Testing	Model extraction defence testing checks whether a model offered through a query interface, such as a prediction API, can be copied by someone who can only send it inputs and read its outputs. The tester plays the attacker, spends a fixed query budget on the protected interface, then uses the answers to train a surrogate model (a copy trained on the answers) or to solve for the model's parameters, and compares the copy with the original. Fidelity is usually measured as the share of inputs on which the copy agrees with the original, together with how many queries that took. The same attack is run with and without each defence, such as limiting queries, rounding or removing confidence scores, or adding noise to outputs, to show how much each defence raises the attacker's cost. The tester also measures the effect on legitimate users, for example accuracy or response time.	Security