Description
Jailbreak resistance testing checks whether a language model or chatbot can be talked out of its safety rules. Testers write or collect prompts that try to elicit content the system is meant to refuse, such as role-playing a character without restrictions, framing the request as hypothetical, hiding it in an encoding such as Base64, or combining several of these. Each response is judged as a refusal, a breach or unclear, by a reviewer or an automated check. The result is the rate at which each attack family succeeds, with the example prompts that worked. This shows where the safeguards fail, but only for the attacks tried.
Example Use Cases
Safety
Testing a mental health support chatbot to ensure it cannot be jailbroken into providing medical advice that contradicts established clinical guidelines or suggesting harmful interventions, even when users employ emotional manipulation or role-playing scenarios.
Testing a legal research assistant used by a law firm to confirm it cannot be pushed, through repeated rewording of a request, into drafting misleading material or giving advice that breaks professional ethics rules.
Security
Validating that a financial advisory AI cannot be manipulated through multi-turn conversations into revealing proprietary trading algorithms, internal risk assessment models, or client portfolio information through social engineering techniques.
Testing an AI marking assistant used by a university to confirm that students cannot talk it into inflating a grade, revealing exam answers or skipping its academic-integrity checks, using role-play or hypothetical framing.
Limitations
- Results describe only the attacks that were tried. New jailbreaks appear constantly, so a clean result today says little about next month, and the attack library has to be refreshed and the tests re-run on a schedule.
- Fixed prompt sets understate a determined attacker. In the original study, trying several attack types on each prompt succeeded on almost every prompt, so coverage depends on the testers' creativity and on how adaptively attacks are chosen.
- Multi-turn jailbreaks build up over a conversation, so they are hard to anticipate and test systematically. A system that refuses single-turn probes can still be steered across several turns.
- Deciding whether a response is a breach is a judgement. Refusals, partial answers and off-topic replies need labelling, outputs can be obfuscated or encoded so manual checking is often needed, and what counts as unacceptable varies by context and culture, so success rates are hard to compare across systems.
- Results hold for one model version, system prompt and safeguard set, and any update can change them. Resistance is also uneven: in the original study one model refused every role-play attack yet remained open to other attack types. And a system that refuses everything looks resistant, so harmless control prompts are needed to tell selective resistance from blanket refusal.
Resources
Research Papers
Jailbroken: How Does LLM Safety Training Fail?
Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.