Chain-of-Thought Faithfulness Evaluation

Description

Chain-of-thought faithfulness evaluation tests whether the step-by-step reasoning a language model writes out is what actually produces its answer, rather than an account added afterwards. One set of tests alters the reasoning itself, by cutting it short, inserting a deliberate mistake, rewording it or replacing it with filler text, and checks whether the final answer changes. If the answer never changes, the reasoning is probably not doing the work. Another set adds a cue to the input that points to a particular answer, such as a user's stated guess, and checks whether the answer shifts while the explanation never mentions the cue. Each test gives a rate, such as how often the answer changes, and passing one rules out a single kind of unfaithfulness without showing that the reasoning is faithful or that its steps are correct.

Example Use Cases

Explainability

An education company's tutoring assistant shows pupils its step-by-step working beside each answer to a science question. Before telling pupils that the working explains the answer, the evaluation team cuts the working short and inserts a deliberate mistake on a test set, and records how often the final answer changes. Where it rarely does, as Lanham et al. found for grade-school science sets, the working is shown as an illustration, not as the reason.

A lender's model-validation team lets an underwriting assistant write a rationale for each recommendation that staff keep on file. It reruns validation cases with a line added saying a senior colleague favours approval. Where the recommendation moves towards the line but the rationale never mentions it, as Turpin et al. found for suggested answers, the rationale is not accepted as the record of why the case was decided.

A public body choosing between two sizes of one language-model family for a casework summarising tool wants staff to rely on the reasoning the model shows. On multiple-choice questions built from its own case files, it records how often each model's answer changes when its reasoning is removed, shuffling answer order to control for letter preference as Bentham et al. did, and prefers the model whose answers depend on its reasoning.

A recruitment platform's audit team uses a language model to compare two candidates for shortlisting when the evidence for each is thin. Following the social-bias test in Turpin et al., it submits each pair twice with the weak evidence swapped between the candidates and counts pairs where the same candidate is picked both times. Those pairs show evidence weighed inconsistently, so the reasoning is not accepted as showing what drove the choice.

Limitations

Passing does not show faithfulness. Each test rules out one way the reasoning could be unfaithful, and the authors call the tests necessary but not sufficient. There is no independent view of the model's actual process to check against, so a clean result supports a narrower claim than 'the reasoning is faithful'.
Results do not carry over between models or tasks. Lanham et al. found that the reasoning changed the final answer under 10% of the time on three science question sets but over 60% on algebra word problems, and that faithfulness varied with model size. Re-measure for the model version and task type that will be deployed.
The answer-change score can reflect something other than faithfulness. Bentham et al. repeated Lanham et al.'s measure on three other model families. After correcting for a model's habit of favouring certain answer letters, apparent unfaithfulness in weaker models fell sharply, and the corrected score tracked accuracy (R-squared of 0.74).
Input-cue tests find only the influences the evaluator thinks to add. Turpin et al. tried three cues, and say their setup only tests faithfulness against minor changes to the input. A model that passes can still be swayed by a cue nobody tried, and the cue must point to a wrong answer to show up as an accuracy drop.
The tests are costly and need control of generation. Lanham et al. sampled 100 reasoning chains per question and re-queried after each truncation; Turpin et al. subsampled for cost. Inserted mistakes were judged plausible in at least 80% of cases, so up to a fifth may test nothing, and Bentham et al. note the procedure needs manual intervention in decoding.

Resources

Research Papers

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Miles Turpin et al.•Dec 9, 2023

Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into LLMs' predictions would yield significant safety benefits. However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction. We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations. When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers. This causes accuracy to drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task, model explanations justify giving answers in line with stereotypes without mentioning the influence of these social biases. Our findings indicate that CoT explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety. Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.

Measuring Faithfulness in Chain-of-Thought Reasoning

Tamera Lanham et al.•Jul 17, 2023

Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.

Software Packages

cot-unfaithfulness (code for Turpin et al. 2023)

Aug 13, 2026

Related Techniques

Name	Description	Assurance Goals
Retrieval-Augmented Generation Evaluation	A retrieval-augmented generation (RAG) system answers a question by retrieving passages from a document collection and asking a language model to write an answer from them. RAG evaluation scores retrieval on whether the fetched passages are relevant and focused and, where reference answers exist, whether they contain what the answer needs. Generation is scored on faithfulness, usually by splitting the answer into individual claims and reporting the share that the fetched passages support, and on answer relevance, which asks whether the answer addresses the question. Frameworks such as Ragas automate these checks with a second language model acting as judge, so faithfulness and relevance need no human-written reference answers. Test sets can also include questions the documents cannot answer, to check that the system says so instead of guessing.	Reliability
Prompt Robustness Testing	Prompt robustness testing measures how much a language model's answers, or its performance on a task, change when the prompt is varied in ways that should not matter, such as rewording, formatting, capitalisation, reordering, or the typos and synonym swaps real users make. The tester starts from task prompts with known correct answers, writes variants that keep the meaning from templates or generates them with perturbation tools such as PromptBench, and runs the model on every variant. Some tools search for the variant that does the most damage, so a drop under that kind of attack is a worst case, not a typical failure rate. The result is reported as the spread of performance across variants, for example the gap between the best and worst format, together with the variants that changed the answer. On one widely used open model, accuracy differed by up to 76 points across formatting changes that kept the meaning, which is why one prompt template is not enough.	Reliability
Few-Shot Fairness Evaluation	Few-shot fairness evaluation tests whether the worked examples placed in a prompt change how fairly a language model treats different demographic groups. The evaluator builds many versions of the same prompt, varying which examples appear and how each demographic group is represented among them. Each version is run on the same test inputs, and group-level results are compared across versions. Typical checks include the gap in outcomes between groups, whether outputs echo stereotypes present in the examples, and whether an input gets the same treatment when only a demographic term in it is swapped. The result shows how sensitive the model's fairness is to the choice of examples, and which compositions make outcomes more or less even.	Fairness