Knowledge Boundary Probing
Description
Knowledge boundary probing maps what a language model can and cannot reliably answer. A probe set is built from questions across the topics the system will face, including some with known answers and some that cannot be answered. Each question is asked in several phrasings, because a model may answer one wording correctly and fail on another. Responses are scored for correctness, for whether the model signals doubt when it should, and for whether its stated or computed confidence tracks whether it is right. The result is a topic-by-topic map of what the model knows, knows only under some phrasings, and does not know, which informs where to deploy it, what to tell users about its limits, and where to add training data or human review.
Example Use Cases
Reliability
A government analytics team using an LLM to summarise policy evidence probes it on well-researched areas and on recent or local policy it cannot have seen in training. Accuracy drops on the second group, so the team adds a retrieval step for those topics before the tool is used in briefings.
A school-software vendor checks an LLM maths tutor against each curriculum unit, asking every question in several wordings. Units where correctness changes with the wording are marked for teacher review instead of being shown to students unsupervised.
Transparency
A bank adds a per-topic table of probe results (accurate on card fees, weak on mortgage rules) to the model documentation for its customer-service chatbot. The app uses the table to tell customers when to check with an adviser.
Limitations
- The probe set samples a finite number of topics and question types. A clean result on those says little about topics the set omits, so the map is only as broad as the probes, and mapping every domain is infeasible.
- Whether a model 'knows' something depends on how the question is worded. A single phrasing can understate or overstate what the model knows, so each question needs several phrasings, and the boundary is fuzzy rather than a clean line.
- Models can be confidently wrong, and can hedge on things they know, so stated or computed confidence is only a rough guide to knowledge. Calibration also varies by topic, so one overall score can hide weak areas.
- Scoring needs labelled questions and a way to detect hedging. Deciding which questions are truly unanswerable takes human judgement, and detectors built on one model's uncertainty phrases may miss how other models express doubt.
- The map holds for one model version. Fine-tuning, a model update or added retrieval shifts the boundary, so the probes must be rerun after each change to keep the map accurate.
Resources
Research Papers
Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation
In recent years, substantial advancements have been made in the development of large language models, achieving remarkable performance across diverse tasks. To evaluate the knowledge ability of language models, previous studies have proposed lots of benchmarks based on question-answering pairs. We argue that it is not reliable and comprehensive to evaluate language models with a fixed question or limited paraphrases as the query, since language models are sensitive to prompt. Therefore, we introduce a novel concept named knowledge boundary to encompass both prompt-agnostic and prompt-sensitive knowledge within language models. Knowledge boundary avoids prompt sensitivity in language model evaluations, rendering them more dependable and robust. To explore the knowledge boundary for a given model, we propose projected gradient descent method with semantic constraints, a new algorithm designed to identify the optimal prompt for each piece of knowledge. Experiments demonstrate a superior performance of our algorithm in computing the knowledge boundary compared to existing methods. Furthermore, we evaluate the ability of multiple language models in several domains with knowledge boundary.
Do Large Language Models Know What They Don’t Know?
Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks.Current research focuses on enhancing their performance within their existing knowledge.Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend.Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance.This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions.We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge.We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts.Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models.Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge.Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge. "True wisdom is knowing what you don't know." -Confucius
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
The notion of uncertainty is of major importance in machine learning and constitutes a key element of machine learning methodology. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments. In particular, this includes the importance of distinguishing between (at least) two different types of uncertainty, often referred to as aleatoric and epistemic . In this paper, we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular.