Few-Shot Fairness Evaluation

Description

Few-shot fairness evaluation tests whether the worked examples placed in a prompt change how fairly a language model treats different demographic groups. The evaluator builds many versions of the same prompt, varying which examples appear and how each demographic group is represented among them. Each version is run on the same test inputs, and group-level results are compared across versions. Typical checks include the gap in outcomes between groups, whether outputs echo stereotypes present in the examples, and whether an input gets the same treatment when only a demographic term in it is swapped. The result shows how sensitive the model's fairness is to the choice of examples, and which compositions make outcomes more or less even.

Example Use Cases

Fairness

Testing whether a resume screening LLM's few-shot examples inadvertently introduce gender bias by showing more male examples for technical positions, affecting how it evaluates subsequent applicants.

Evaluating whether a medical triage LLM's few-shot examples for symptom assessment inadvertently encode demographic biases (e.g., more examples of cardiac symptoms in male patients), leading to differential urgency assessments across patient populations.

Assessing whether few-shot examples in a loan application review assistant systematically present more favourable examples for certain demographic profiles, biasing the model's assessment of creditworthiness for subsequent applications.

Testing whether an automated essay grading system keeps consistent standards across student demographics when its few-shot examples over-represent particular writing styles, dialects or cultural references.

Limitations

The number of possible example sets, orderings and wordings is far too large to test exhaustively. The evaluation samples configurations and can miss the combinations that cause the worst disparities, so results describe what was tested rather than every prompt the system will meet.
Fairness measures can shift with small changes in example wording, order or prompt format, and with ordinary sampling variation in the model's replies. A gap between two example sets may be noise unless each is run repeatedly, and results seldom carry over to a different prompt template, instruction style or model version.
Needs test inputs labelled with demographic attributes, or pairs of inputs that differ only in a demographic term. These can be unavailable, costly to build or sensitive to hold, and they must reflect how the system will actually be prompted, which takes domain expertise to judge.
Each extra example set multiplies the number of model calls, because every set must be run on enough inputs for each group being compared. With large models and high inference costs, thorough coverage is expensive, which tempts teams to test fewer configurations.

Resources

Research Papers

Fairness-guided few-shot prompting for large language models

Huan Ma et al.•Jan 1, 2023

Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper, we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner.

Software Packages

langfair

Sep 11, 2026

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

Tutorials

LangFair counterfactual metrics demo notebook

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments - cvs-health/langfair

Documentations

LangFair documentation

Related Techniques

Name	Description	Assurance Goals
Causal Discrimination Testing	Causal discrimination testing checks whether a system gives different outputs to otherwise identical inputs because of a protected attribute, by changing only that attribute, or a name, term or other proxy that signals it, and measuring how often or how much the output changes. Galhotra, Brun and Meliou (2017), who called the approach fairness testing, generate inputs from a description of the valid input fields, make copies of each that differ only in the protected attribute (for example, recorded sex), run them all through the system as a black box, and report the causal discrimination score: the share of inputs whose decision changes, sampled until it is known within a chosen margin of error and confidence level. For text, the same test substitutes identity terms or names in otherwise unchanged sentences, such as one religion or ethnicity for another, and reports the average change in the model's score across the substitutions, which Garg et al. (2019) call the counterfactual token fairness gap. Because every other part of the input is held fixed, the test needs no model of how the attribute influences other variables: it measures whether the system itself responds to the attribute, or to the chosen proxy. The output is a score per attribute and, if the tester logs them, the input pairs whose outputs differed.	Fairness
Equal Opportunity Difference	Equal Opportunity Difference checks whether a model is equally likely to give a favourable prediction to the people whose true outcome is favourable, such as loan applicants who would repay, in each demographic group. For each group it calculates the true positive rate: of the people whose true outcome is positive, the share the model also predicts as positive. The score is the gap between two groups' rates, usually the unprivileged group's minus the privileged group's, and 0 means the rates match. Some libraries report the largest gap across all groups instead. The metric follows the equality of opportunity criterion of Hardt, Price and Srebro (2016), and it says nothing about people whose true outcome is negative, so a score of 0 does not show that the model is fair overall.	Fairness
Embedding Bias Analysis	Embedding bias analysis tests whether the numerical representations a model uses for words, images or documents link social groups to stereotyped attributes. An embedding places each item at a point in a many-dimensional space so that similar items sit close together. The best-known test, the Word Embedding Association Test (Caliskan et al., 2017), compares how close two sets of target words, such as engineering and nursing occupations, sit to two sets of attribute words, such as male and female terms, and reports an effect size and a significance level. Other analyses project embeddings onto a gender or ethnicity direction, or cluster them to see whether groups separate. The scores describe the representation itself, and any link to bias in an application built on it has to be tested separately.	Fairness