Embedding Bias Analysis

Description

Embedding bias analysis tests whether the numerical representations a model uses for words, images or documents link social groups to stereotyped attributes. An embedding places each item at a point in a many-dimensional space so that similar items sit close together. The best-known test, the Word Embedding Association Test (Caliskan et al., 2017), compares how close two sets of target words, such as engineering and nursing occupations, sit to two sets of attribute words, such as male and female terms, and reports an effect size and a significance level. Other analyses project embeddings onto a gender or ethnicity direction, or cluster them to see whether groups separate. The scores describe the representation itself, and any link to bias in an application built on it has to be tested separately.

Example Use Cases

Fairness

A team building a CV-screening tool must choose between two pretrained word embedding sets. It runs association tests on each, comparing how close male and female terms sit to career and family words, and drops or mitigates the set with the larger stereotyped association before building on it.

Analysing clinical note embeddings in a medical diagnosis support system to detect whether disease associations cluster along demographic lines (for example, certain conditions being systematically closer to particular age or ethnic groups in the embedding space), which could lead to biased treatment recommendations.

Auditing loan application embeddings to detect whether protected characteristics (ethnicity, gender, age) cluster near creditworthiness concepts in ways that could enable indirect discrimination in lending decisions.

Limitations

Bias found in an embedding space does not reliably predict bias in the systems built on it. Goldfarb-Tarrant et al. (2021) found no consistent correlation between association-test scores and gaps between groups in downstream tasks, so a clean or a biased result here needs a separate test on the application itself.
Results depend on the word or sentence lists chosen to represent each group and attribute. Swapping one published list of pleasant words for another moved the effect size in the original study from 1.50 to 1.28, so a single list can overstate or understate bias, and lists written in one culture or language may not transfer.
A passing score does not show that bias is absent. Tests and projections look only for the pattern they are built to find, and Gonen and Goldberg (2019) showed that gender information removed by projection could still be recovered from the distances between words.
There is no agreed threshold for an acceptable association. An effect size describes how strongly two groups of terms differ in closeness, not how much harm follows, so what counts as a pass depends on the application and has to be set and justified by the team.
Needs the embedding vectors themselves, so it cannot be run on a system that exposes only final decisions or text. Pretrained vectors can be tested without the training data, as the original study did with published GloVe vectors.

Resources

Research Papers

Semantics derived automatically from language corpora contain human-like biases

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan•Apr 14, 2017

Machines learn what people know implicitly AlphaGo has demonstrated that a machine can learn how to do things that people spend many years of concentrated study learning, and it can rapidly learn how to do them better than any human can. Caliskan et al. now show that machines can learn word associations from written texts and that these associations mirror those learned by humans, as measured by the Implicit Association Test (IAT) (see the Perspective by Greenwald). Why does this matter? Because the IAT has predictive value in uncovering the association between concepts, such as pleasantness and flowers or unpleasantness and insects. It can also tease out attitudes and beliefs—for example, associations between female names and family or male names and career. Such biases may not be expressed explicitly, yet they can prove influential in behavior. Science , this issue p. 183 ; see also p. 133

Software Packages

WEFE: The Word Embeddings Fairness Evaluation Framework

Oct 6, 2026

WEFE: The Word Embeddings Fairness Evaluation Framework. WEFE is a framework that standardizes the bias measurement and mitigation in Word Embeddings models. Please feel welcome to open an issue in case you have any questions or a pull request if you want to contribute to the project!

Tutorials

Bias Measurement — WEFE user guide

Documentations

WEFE documentation

Related Techniques

Name	Description	Assurance Goals
Few-Shot Fairness Evaluation	Few-shot fairness evaluation tests whether the worked examples placed in a prompt change how fairly a language model treats different demographic groups. The evaluator builds many versions of the same prompt, varying which examples appear and how each demographic group is represented among them. Each version is run on the same test inputs, and group-level results are compared across versions. Typical checks include the gap in outcomes between groups, whether outputs echo stereotypes present in the examples, and whether an input gets the same treatment when only a demographic term in it is swapped. The result shows how sensitive the model's fairness is to the choice of examples, and which compositions make outcomes more or less even.	Fairness
Bias Benchmark for Question Answering	The Bias Benchmark for Question Answering (BBQ) is a fixed set of about 58,500 multiple-choice reading questions that tests whether a language model's answers rely on social stereotypes, across nine social dimensions (age, disability status, gender identity, nationality, physical appearance, race and ethnicity, religion, socio-economic status and sexual orientation) and two combinations of them, using stereotypes documented as harmful in US English-speaking contexts (Parrish et al., 2022). Each item describes two people from different groups and asks a question that points at a stereotype, such as who is bad at maths, with three answers to choose from: either person, or an answer such as 'cannot be determined'; in the ambiguous version the passage does not say which person the question is about, so 'cannot be determined' is correct, and in the disambiguated version an added sentence settles it. Items come in balanced sets that swap the two groups and their order and pair each negative question with a non-negative one, so that a model which simply prefers one group whatever the question is not counted as stereotyping. From the model's chosen answers (generated, or picked by comparing the probabilities it gives the three options) the test reports accuracy in each version and a bias score per category from -100% to 100%: the share of answers naming a person that follow the stereotype, rescaled so that 0 means no consistent direction and a negative score leans against the stereotype, and in the ambiguous version multiplied by the error rate, so that stereotyped guesses weigh more when the model guesses often; it also reports how far accuracy falls when the correct answer goes against the stereotype. Unlike substituting identity terms into a deployer's own inputs, or varying the people shown in few-shot examples, BBQ is a fixed external question set that tests what a model concludes about people when the text does, and does not, support a conclusion.	Fairness
Causal Discrimination Testing	Causal discrimination testing checks whether a system gives different outputs to otherwise identical inputs because of a protected attribute, by changing only that attribute, or a name, term or other proxy that signals it, and measuring how often or how much the output changes. Galhotra, Brun and Meliou (2017), who called the approach fairness testing, generate inputs from a description of the valid input fields, make copies of each that differ only in the protected attribute (for example, recorded sex), run them all through the system as a black box, and report the causal discrimination score: the share of inputs whose decision changes, sampled until it is known within a chosen margin of error and confidence level. For text, the same test substitutes identity terms or names in otherwise unchanged sentences, such as one religion or ethnicity for another, and reports the average change in the model's score across the substitutions, which Garg et al. (2019) call the counterfactual token fairness gap. Because every other part of the input is held fixed, the test needs no model of how the attribute influences other variables: it measures whether the system itself responds to the attribute, or to the chosen proxy. The output is a score per attribute and, if the tester logs them, the input pairs whose outputs differed.	Fairness