Constitutional AI Evaluation
Description
Constitutional AI evaluation checks whether a language model follows the written principles, or 'constitution', that it was trained or instructed to follow. The evaluator writes test prompts for each principle, and can add prompts where two principles conflict or where a user tries to talk the model out of a principle. A judge, either human reviewers or a second language model whose accuracy has been checked, assesses each response against one principle at a time, often by comparing it with a baseline model's response. The result is a rate for each principle, such as how often the model's response is preferred to the baseline's, which shows which principles it follows well on the prompts tested and which it does not. A fuller evaluation also checks that following the principles has not made the model over-cautious or less capable.
Example Use Cases
Safety
A developer tuning a general-purpose assistant on a constitution that includes 'refuse requests for illegal activity' writes test prompts for each principle, including borderline chemistry-education questions and attempts to extract synthesis instructions. Reviewers compare the tuned model's answers with the base model's, to decide whether the constitution is ready to ship or needs principles rewritten.
A digital health provider tests a mental health support chatbot built on the principle 'be supportive, but never diagnose' with scripted crisis conversations in which users ask directly for a diagnosis. Clinicians score each reply, and the share that stays on the right side of the boundary decides whether the chatbot is cleared for a limited pilot.
Reliability
A news publisher using an AI summariser governed by a neutrality principle gives it the same contested story framed from several political positions. Reviewers from differing viewpoints score whether each summary adds editorial language, and the editors use the rate by topic to decide where the summariser can run without a human check.
An education company checks a homework assistant built on the principle 'guide learning without giving direct answers' across subjects, age bands and frustrated or urgent requests. It counts replies that simply give the answer, and uses the rate for each subject to decide where the principle needs strengthening before a school rollout.
Limitations
- Principles are written in natural language, so compliance depends on how the judge reads them. Vague principles, or ones that mean different things across cultures and languages, give scores that reviewers would dispute, so the judge's reading needs checking against human reviewers for each principle.
- Principles can conflict in complex scenarios, and the evaluation has to decide which should win. If the intended priority is not written down, a model can look compliant on each principle separately and still resolve conflicts differently from what the developer intended.
- Adherence is uneven across kinds of principle. In one study of small fine-tuned models, negatively framed principles such as avoiding harmful content were followed much better than positively framed, abstract ones such as benefiting humanity, so an average score can hide principles the model largely ignores.
- Scores depend on the test prompts and on the judge. Prompts that miss a principle's edge cases give a falsely high rate, and an automated judge can favour one response position or misapply the principle, so it needs checking against human judgement in each new domain.
- When principles are revised after deployment, results must be re-run, and comparing them across principle versions is hard because a change in score may come from the model, the principles or the test criteria.
Resources
Research Papers
C3AI: Crafting and Evaluating Constitutions for Constitutional AI
Constitutional AI (CAI) guides LLM behavior using constitutions, but identifying which principles are most effective for model alignment remains an open challenge. We introduce the C3AI framework (Crafting Constitutions for CAI models), which serves two key functions: (1) selecting and structuring principles to form effective constitutions before fine-tuning; and (2) evaluating whether fine-tuned CAI models follow these principles in practice. By analyzing principles from AI and psychology, we found that positively framed, behavior-based principles align more closely with human preferences than negatively framed or trait-based principles. In a safety alignment use case, we applied a graph-based principle selection method to refine an existing CAI constitution, improving safety measures while maintaining strong general reasoning capabilities. Interestingly, fine-tuned CAI models performed well on negatively framed principles but struggled with positively framed ones, in contrast to our human alignment results. This highlights a potential gap between principle design and model adherence. Overall, C3AI provides a structured and scalable approach to both crafting and evaluating CAI constitutions.
Constitutional AI: Harmlessness from AI Feedback
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.