Few-Shot Fairness Evaluation
Description
Few-shot fairness evaluation tests whether the worked examples placed in a prompt change how fairly a language model treats different demographic groups. The evaluator builds many versions of the same prompt, varying which examples appear and how each demographic group is represented among them. Each version is run on the same test inputs, and group-level results are compared across versions. Typical checks include the gap in outcomes between groups, whether outputs echo stereotypes present in the examples, and whether an input gets the same treatment when only a demographic term in it is swapped. The result shows how sensitive the model's fairness is to the choice of examples, and which compositions make outcomes more or less even.
Example Use Cases
Fairness
Testing whether a resume screening LLM's few-shot examples inadvertently introduce gender bias by showing more male examples for technical positions, affecting how it evaluates subsequent applicants.
Evaluating whether a medical triage LLM's few-shot examples for symptom assessment inadvertently encode demographic biases (e.g., more examples of cardiac symptoms in male patients), leading to differential urgency assessments across patient populations.
Assessing whether few-shot examples in a loan application review assistant systematically present more favourable examples for certain demographic profiles, biasing the model's assessment of creditworthiness for subsequent applications.
Testing whether an automated essay grading system keeps consistent standards across student demographics when its few-shot examples over-represent particular writing styles, dialects or cultural references.
Limitations
- The number of possible example sets, orderings and wordings is far too large to test exhaustively. The evaluation samples configurations and can miss the combinations that cause the worst disparities, so results describe what was tested rather than every prompt the system will meet.
- Fairness measures can shift with small changes in example wording, order or prompt format, and with ordinary sampling variation in the model's replies. A gap between two example sets may be noise unless each is run repeatedly, and results seldom carry over to a different prompt template, instruction style or model version.
- Needs test inputs labelled with demographic attributes, or pairs of inputs that differ only in a demographic term. These can be unavailable, costly to build or sensitive to hold, and they must reflect how the system will actually be prompted, which takes domain expertise to judge.
- Each extra example set multiplies the number of model calls, because every set must be run on enough inputs for each group being compared. With large models and high inference costs, thorough coverage is expensive, which tempts teams to test fewer configurations.
Resources
Research Papers
Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning
Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models (LLMs) in processing tabular data, a challenging task given the structured nature of such data.Despite advancements in performance, the fairness implications of these methods are less understood.This study investigates how varying demonstrations within ICL prompts influence the fairness outcomes of LLMs.Our findings reveal that deliberately including minority group samples in prompts significantly boosts fairness without sacrificing predictive accuracy.Further experiments demonstrate that the proportion of minority to majority samples in demonstrations affects the trade-off between fairness and prediction accuracy.Based on these insights, we introduce a mitigation technique that employs clustering and evolutionary strategies to curate a diverse and representative sample set from the training data.This approach aims to enhance both predictive performance and fairness in ICL applications.Experimental results validate that our proposed method dramatically improves fairness across various metrics, showing its efficacy in real-world scenarios.