Agent Goal Misalignment Testing

Description

Agent goal misalignment testing checks whether an AI agent takes unsanctioned means to reach a goal it has been given. The tester builds scenarios in which the agent has a task, tools and access, and a forbidden route to its goal is open, such as deceiving a user, leaking confidential information or coercing people to avoid being replaced. Each scenario is run many times, and each transcript is classified as misaligned or not, either by a language-model classifier checked against human judgement or by human annotators. The result is the rate of misaligned actions per scenario and per model, with example transcripts, and how that rate changes when the tester varies a factor such as a threat to the agent's continued operation or an explicit instruction not to cause harm. The test needs no access to a reward function or to training, because it examines the behaviour of an already-trained agent.

Example Use Cases

Safety

Before giving an email and calendar agent access to client records, a bank runs it through scenarios in which its assigned target can only be met by sending confidential client data to an outside party, and records how often it does so with and without a system-prompt prohibition, to decide whether the agent needs a human approval step for outbound messages.

A national AI evaluation body tests coding agents deployed inside a simulated research lab for whether they sabotage or refuse safety-related tasks when the scenario says the agent is about to be replaced, and reports the rates to the developers before wider internal deployment.

Reliability

A logistics firm upgrading the model behind its procurement agent runs the old and new models through the same scenarios, in which a delivery deadline can only be met by exceeding the agent's delegated spending limit or misreporting an order's status, and adopts the new model only if it stays within its operator's constraints at least as often.

Limitations

Scenarios are artificial. To make behaviour measurable they usually offer a stark choice between failing the goal and a harmful act, and place the key information side by side, whereas real deployments offer more alternatives. Measured rates may therefore overstate or understate what an agent would do in use, and do not transfer directly to a deployment.
Agents may behave differently when they judge they are being tested. In the agentic-misalignment study one model blackmailed in 55.1% of runs when it concluded the scenario was real and 6.5% when it concluded it was an evaluation, so a low measured rate can understate deployed behaviour.
It only tests the misaligned routes the scenarios offer. A clean result on one suite says nothing about routes it did not include, and building enough realistic scenario variants for a given deployment (tools, data, organisational context) takes substantial evaluation-design and domain effort.
Rates are only as good as the transcript classification. Classifiers based on language models, or keyword matching, misjudge borderline actions, and their error rate feeds straight into the reported rates; a sample of transcripts needs human checking for each new scenario family.
It measures how often an agent acts, not why. An agent's stated reasoning may not reflect what drove its action, and role-play can explain some behaviour, so transcripts cannot establish whether a misaligned action came from the goal, the framing or the scenario's cues.

Resources

Research Papers

Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark

Alexander Pan et al.•Apr 6, 2023

Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.

Software Packages

agentic-misalignment (anthropic-experimental)

Oct 7, 2026

Documentations

Agentic Misalignment: How LLMs could be insider threats

Jun 20, 2025

New research on simulated blackmail, industrial espionage, and other misaligned behaviors in LLMs

Related Techniques

Name	Description	Assurance Goals
Reward Hacking Detection	Reward hacking is when an agent trained to maximise a proxy reward (a score that only approximates what its designers want) raises that score by exploiting its gaps while the intended outcome stalls or worsens. Detection scores each trained policy (the agent's decision rule) on the proxy and on an independent measure of the intended objective that training never used, such as a true reward from a simulator, a held-out 'gold' reward model or human ratings. Repeating this at every saved training checkpoint and model size matters because the originating study found capability thresholds at which the true reward drops sharply while the proxy keeps rising. When the true objective cannot be scored in operation, the study instead flags a policy whose actions diverge too far from those of a trusted policy that people have already verified. The output is a pair of reward curves showing where they part, or an anomaly score for each candidate policy.	Reliability Safety
Dangerous Capability Evaluation	Dangerous capability evaluation measures whether a frontier AI model can carry out tasks that could enable severe harm, such as finding and exploiting software vulnerabilities, persuading or deceiving people, acquiring resources and running copies of itself, or helping to plan an attack with chemical, biological, radiological or nuclear materials, and compares the results with capability thresholds that the developer's safety framework fixes before testing. Evaluators first write a threat model for each risk domain; the framework names a critical capability level, at which the model could pose an extreme risk without extra safeguards, and an early-warning trigger defined on the evaluation results, which the method's originators sketched for one domain as success on more than half of a set of tasks given ten attempts at each. They then build task suites for each domain, such as capture-the-flag hacking challenges in sandboxed networks or multi-step tasks that require the model to set up a cloud server and install software, and try to draw out the model's best performance: it is tested without the safety filters of the deployed product, given agent scaffolding (prompts, planning loops and tools such as a command shell) and sometimes fine-tuned, because the evaluation must not underestimate what the model can do. Scoring records success over repeated attempts and, for tasks the model cannot finish, how close it came, by starting it from intermediate milestones and measuring how much expert help it needs to succeed. Unlike red teaming, which probes a system and its safeguards for whatever failures can be found, and unlike tests of whether a model refuses harmful requests, it measures what the model is able to do, and whether the results cross the trigger informs decisions on deployment and on the security measures the model needs.	Safety
Multi-Agent System Testing	Multi-agent system testing examines systems in which several AI agents act on shared tasks or resources, to find failures that no single agent shows on its own. Testers run the agents together in simulated or scripted scenarios, often with injected faults such as failed sensors, delayed messages or conflicting goals, and record the failures that appear. Checks cover whether the task is completed, whether coordination breaks down (for example through deadlock, collisions or one agent's error spreading to others), and whether stated safety rules are ever broken. For language-model agents, benchmarks such as MultiAgentBench score task completion and the quality of collaboration or competition. For small or abstract models, a model checker such as MCMAS can search every reachable state for a violation of a rule written in formal logic and, for many kinds of rule, return an example run that breaks it.	Safety Reliability