Prompt Robustness Testing
Description
Prompt robustness testing measures how much a language model's answers, or its performance on a task, change when the prompt is varied in ways that should not matter, such as rewording, formatting, capitalisation, reordering, or the typos and synonym swaps real users make. The tester starts from task prompts with known correct answers, writes variants that keep the meaning from templates or generates them with perturbation tools such as PromptBench, and runs the model on every variant. Some tools search for the variant that does the most damage, so a drop under that kind of attack is a worst case, not a typical failure rate. The result is reported as the spread of performance across variants, for example the gap between the best and worst format, together with the variants that changed the answer. On one widely used open model, accuracy differed by up to 76 points across formatting changes that kept the meaning, which is why one prompt template is not enough.
Example Use Cases
Reliability
A digital health provider testing a symptom-triage chatbot rewrites each benchmark scenario as a brief layperson message, a detailed account, a clinician's summary and an anxious message, and checks that the urgency category stays the same across all four, treating any change as a failure to investigate before release.
A public-sector team choosing between candidate language models for classifying citizen enquiries evaluates each across a set of plausible prompt formats and instruction paraphrases, and compares models on worst-format and average accuracy rather than on a single prompt, because the ranking of models can change with the template.
A bank validating a model that summarises suspicious-transaction reports checks that the risk category it extracts does not change when the same activity is described with different terminology, abbreviations or layout, as reports from different branches would be.
Limitations
- The space of possible prompt variants is unbounded, so a test samples it. A model can pass on every variant tried and still fail on a phrasing nobody generated, and the conclusion holds only for the kinds of variation the test set covers.
- Whether a variant keeps the meaning of the original is a judgement. Automated perturbations can change meaning or produce text no user would write, and attack-based tools search for the most damaging variant, so their drops are worst cases. Variant sets need checking by someone who knows the domain.
- Results do not transfer between models: which formats work well correlated only weakly across models in a large study, and robustness to one domain's phrasing says little about another's. The test has to be rerun for each model version, each prompt template change and each domain.
- Every variant multiplies the number of model calls, and for open-ended answers each response may need scoring by a domain expert rather than against a fixed label. Testing many formats across many examples and candidate models quickly becomes costly.
- It shows that outputs are brittle, not why. Choosing the best-scoring template on the same variants used to test it gives an optimistic estimate of how that template will perform on new inputs, so template selection and the final robustness measurement need separate data.
Resources
Research Papers
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
The increasing reliance on Large Language Models (LLMs) across academia and industry necessitates a comprehensive understanding of their robustness to prompts. In response to this vital need, we introduce PromptRobust, a robustness benchmark designed to measure LLMs' resilience to adversarial prompts. This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts, crafted to mimic plausible user errors like typos or synonyms, aim to evaluate how slight deviations can affect LLM outcomes while maintaining semantic integrity. These prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our study generates 4, 788 adversarial prompts, meticulously evaluated over 8 tasks and 13 datasets. Our findings demonstrate that contemporary LLMs are not robust to adversarial prompts. Furthermore, we present comprehensive analysis to understand the mystery behind prompt robustness and its transferability. We then offer insightful robustness analysis, beneficial to both researchers and everyday users. Code is available at https://github.com/microsoft/promptbench.
State of What Art? A Call for Multi-Prompt LLM Evaluation
Recent advances in LLMs have led to an abundance of evaluation benchmarks, which typically rely on a single instruction template per task. We create a large-scale collection of instruction paraphrases and comprehensively analyze the brittleness introduced by single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. We find that different instruction templates lead to very different performance, both absolute and relative. Instead, we propose a set of diverse metrics on multiple instruction paraphrases, specifically tailored for different use cases (e.g., LLM vs. downstream development), ensuring a more reliable and meaningful assessment of LLM capabilities. We show that our metrics provide new insights into the strengths and limitations of current LLMs.
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
As large language models (LLMs) are adopted as a fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format. We also show that format performance only weakly correlates between models, which puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FormatSpread, an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights. Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.