Retrieval-Augmented Generation Evaluation
Description
A retrieval-augmented generation (RAG) system answers a question by retrieving passages from a document collection and asking a language model to write an answer from them. RAG evaluation scores retrieval on whether the fetched passages are relevant and focused and, where reference answers exist, whether they contain what the answer needs. Generation is scored on faithfulness, usually by splitting the answer into individual claims and reporting the share that the fetched passages support, and on answer relevance, which asks whether the answer addresses the question. Frameworks such as Ragas automate these checks with a second language model acting as judge, so faithfulness and relevance need no human-written reference answers. Test sets can also include questions the documents cannot answer, to check that the system says so instead of guessing.
Example Use Cases
Reliability
A software company's support team tests its troubleshooting assistant on a set of real customer questions. For each answer it checks that the retrieved pages are the right ones and that every configuration step or command quoted appears in the official documentation, and it holds back changes that introduce invented steps.
A law firm's research team tests an assistant that summarises case law from its own database. Each summary is split into claims and each claim is checked against the passages the assistant retrieved, and summaries with unsupported claims are flagged for review before the assistant is rolled out to fee-earners.
A hospital pharmacy team tests a clinical decision-support tool that retrieves guidelines and patient records. A clinician compares each generated treatment suggestion with the retrieved guideline text, and the team tracks the share of suggestions that go beyond what the sources say before the tool reaches ward staff.
A benefits agency tests its eligibility chatbot on questions drawn from real enquiries, including some the policy documents cannot answer. It checks that quoted amounts and rules match the retrieved policy text and that the chatbot says it cannot answer instead of guessing, before the chatbot is opened to citizens.
Limitations
- Scores that judge retrieval against known answers, such as context recall, need reference answers or labelled passages, which are slow to write and need experts in specialist domains. Reference-free scores avoid this but only penalise irrelevant passages, so they cannot show that a needed passage was never retrieved.
- Scores usually come from a second language model acting as judge, so they inherit its errors and its sensitivity to wording. In the original Ragas study the judge agreed with human raters in 95% of pairwise comparisons on faithfulness but only 70% on context relevance, and it struggled to pick out the key sentences in long contexts.
- Faithfulness measures whether an answer is supported by the retrieved text, not whether that text is correct, current or complete. A system that retrieves an outdated document and reports it accurately scores well, so the knowledge base needs checking separately.
- Claim-by-claim checking may not catch answers that chain reasoning across several documents. Misleading emphasis, selective quotation and unsupported leaps between sources can pass when each sentence is individually supported, so these cases need targeted test questions and human review.
- When a score is low, the cause can lie in retrieval, in generation or in how the two interact, and the scores alone often cannot say which. Diagnosis usually needs manual inspection of the retrieved passages and answers for a sample of failing questions.
Resources
Research Papers
RAGAs: Automated Evaluation of Retrieval Augmented Generation
We introduce RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAGAs is available at [https://github.com/explodinggradients/ragas]. RAG systems are composed of a retrieval and an LLM based generation module. They provide LLMs with knowledge from a reference textual database, enabling them to act as a natural language layer between a user and textual databases, thus reducing the risk of hallucinations. Evaluating RAG architectures is challenging due to several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages faithfully, and the quality of the generation itself. With RAGAs, we introduce a suite of metrics that can evaluate these different dimensions without relying on ground truth human annotations. We posit that such a framework can contribute crucially to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.