Dangerous Capability Evaluation

Description

Dangerous capability evaluation measures whether a frontier AI model can carry out tasks that could enable severe harm, such as finding and exploiting software vulnerabilities, persuading or deceiving people, acquiring resources and running copies of itself, or helping to plan an attack with chemical, biological, radiological or nuclear materials, and compares the results with capability thresholds that the developer's safety framework fixes before testing. Evaluators first write a threat model for each risk domain; the framework names a critical capability level, at which the model could pose an extreme risk without extra safeguards, and an early-warning trigger defined on the evaluation results, which the method's originators sketched for one domain as success on more than half of a set of tasks given ten attempts at each. They then build task suites for each domain, such as capture-the-flag hacking challenges in sandboxed networks or multi-step tasks that require the model to set up a cloud server and install software, and try to draw out the model's best performance: it is tested without the safety filters of the deployed product, given agent scaffolding (prompts, planning loops and tools such as a command shell) and sometimes fine-tuned, because the evaluation must not underestimate what the model can do. Scoring records success over repeated attempts and, for tasks the model cannot finish, how close it came, by starting it from intermediate milestones and measuring how much expert help it needs to succeed. Unlike red teaming, which probes a system and its safeguards for whatever failures can be found, and unlike tests of whether a model refuses harmful requests, it measures what the model is able to do, and whether the results cross the trigger informs decisions on deployment and on the security measures the model needs.

Example Use Cases

Safety

A frontier AI developer runs its cyber-offence task suite on each major checkpoint during training and before release, using the agent scaffold its evaluation team has found strongest. Its safety framework sets an early-warning trigger on the share of professional-level capture-the-flag challenges solved; a checkpoint that crosses it cannot be released until stronger protection of the model weights and mitigations against cyber misuse are in place.

A national AI security institute given access to a model before release runs autonomy tasks, such as setting up a cloud server and installing an open-source model on it, both end to end and from intermediate milestones, with a staff member approving each action the agent takes on the live internet. It reports to the developer how close the model came to completing each task and how this compares with the developer's previous model.

A company preparing to publish a model's weights openly evaluates a version fine-tuned on task-relevant data as well as the version it plans to release, because anyone who downloads the weights can fine-tune them, and compares the fine-tuned version's results with its threshold when deciding whether to publish.

Limitations

A result below the threshold is only as strong as the effort spent drawing out the model's capability. Better scaffolds, more attempts, fine-tuning or a later checkpoint can raise scores, and models can be prompted or fine-tuned to underperform deliberately on dangerous-capability tasks while scoring normally elsewhere, so a negative result is a lower bound on capability, not proof of its absence.
The link between a task score and real-world harm is weakly grounded. The method's originators describe their early-warning trigger as early thinking in an extremely immature area and set no thresholds for their chemical, biological, radiological and nuclear evaluations, where most scientific information is dual-use, so different developers can draw the line in different places and reach different decisions from the same scores.
Task suites cover only the threat models someone has written down, and laboratory tasks may not reflect real conditions: persuasion measured in short dialogues with paid participants, or hacking measured on capture-the-flag challenges, may overstate or understate what the model could do against real people and real systems.
Agentic tasks need sandboxed environments, many repeated runs and expert grading, and tasks with real internet access need a person to approve each action so the agent cannot cause harm, which makes full runs slow and costly. Testing the unfiltered model also needs access that external evaluators may not have; testing the deployed product with its safety filters measures something different.
Results date quickly. Scores change with each new checkpoint and fine-tuning recipe, so evaluations have to be repeated for every significant model update, and published tasks can leak into training data, which is why the originating team withheld solutions and flag values from the tasks it released.

Resources

Research Papers

Evaluating Frontier Models for Dangerous Capabilities

Mary Phuong et al.•Apr 5, 2024

To understand the risks posed by a new AI system, we must understand what it can and cannot do. Building on prior work, we introduce a programme of new "dangerous capability" evaluations and pilot them on Gemini 1.0 models. Our evaluations cover four areas: (1) persuasion and deception; (2) cyber-security; (3) self-proliferation; and (4) self-reasoning. We do not find evidence of strong dangerous capabilities in the models we evaluated, but we flag early warning signs. Our goal is to help advance a rigorous science of dangerous capability evaluation, in preparation for future models.

Software Packages

Inspect Evals

Collection of evals for Inspect AI. Contribute to UKGovernmentBEIS/inspect_evals development by creating an account on GitHub.

Tutorials

Inspect tutorial: Agent Evals (capture the flag)

Open-source framework for large language model evaluations

Documentations

Self-proliferation Tasks (GDM Dangerous Capabilities) — Inspect Evals documentation

Ten real-world–inspired tasks from Google DeepMind’s Dangerous Capabilities Evaluations assessing self-proliferation behaviors (e.g., email setup, model installation, web agent setup, wallet operations). Supports end-to-end, milestones, and expert best-of-N modes.

Related Techniques

Name	Description	Assurance Goals
Agent Goal Misalignment Testing	Agent goal misalignment testing checks whether an AI agent takes unsanctioned means to reach a goal it has been given. The tester builds scenarios in which the agent has a task, tools and access, and a forbidden route to its goal is open, such as deceiving a user, leaking confidential information or coercing people to avoid being replaced. Each scenario is run many times, and each transcript is classified as misaligned or not, either by a language-model classifier checked against human judgement or by human annotators. The result is the rate of misaligned actions per scenario and per model, with example transcripts, and how that rate changes when the tester varies a factor such as a threat to the agent's continued operation or an explicit instruction not to cause harm. The test needs no access to a reward function or to training, because it examines the behaviour of an already-trained agent.	Safety Reliability
Reward Hacking Detection	Reward hacking is when an agent trained to maximise a proxy reward (a score that only approximates what its designers want) raises that score by exploiting its gaps while the intended outcome stalls or worsens. Detection scores each trained policy (the agent's decision rule) on the proxy and on an independent measure of the intended objective that training never used, such as a true reward from a simulator, a held-out 'gold' reward model or human ratings. Repeating this at every saved training checkpoint and model size matters because the originating study found capability thresholds at which the true reward drops sharply while the proxy keeps rising. When the true objective cannot be scored in operation, the study instead flags a policy whose actions diverge too far from those of a trusted policy that people have already verified. The output is a pair of reward curves showing where they part, or an anomaly score for each candidate policy.	Reliability Safety
Jailbreak Resistance Testing	Jailbreak resistance testing checks whether a language model or chatbot can be talked out of its safety rules. Testers write or collect prompts that try to elicit content the system is meant to refuse, such as role-playing a character without restrictions, framing the request as hypothetical, hiding it in an encoding such as Base64, or combining several of these. Each response is judged as a refusal, a breach or unclear, by a reviewer or an automated check. The result is the rate at which each attack family succeeds, with the example prompts that worked. This shows where the safeguards fail, but only for the attacks tried.	Safety Security