Fairness
Bias Measurement
Techniques that test for and quantify unequal treatment linked to protected attributes, in outputs, inputs or representations.
15 techniques
15 techniques
| Goals | Models | Data Types | Description | |||
|---|---|---|---|---|---|---|
| Permutation Tests | Algorithmic | Architecture/model Agnostic Requirements/black Box | Any | A permutation test asks how likely an observed result would be if there were no real relationship in the data. The... | ||
| Demographic Parity Assessment | Metric | Architecture/model Agnostic Paradigm/supervised +1 | Any | Demographic Parity Assessment checks whether a model gives a favourable outcome, such as a loan approval or an interview... | ||
| Counterfactual Fairness Assessment | Testing | Architecture/model Agnostic Paradigm/supervised +1 | Tabular | Counterfactual Fairness Assessment tests whether a model would have made the same predictions for a person had their... | ||
| Causal Discrimination Testing | Testing | Architecture/model Agnostic Paradigm/supervised +1 | Tabular Text | Causal discrimination testing checks whether a system gives different outputs to otherwise identical inputs because of a... | ||
| Red Teaming | Process | Architecture/model Agnostic Requirements/black Box | Any | Red teaming is a structured exercise in which a team deliberately tries to make an AI system fail, so that weaknesses... | ||
| Equal Opportunity Difference | Metric | Architecture/model Agnostic Paradigm/supervised +1 | Any | Equal Opportunity Difference checks whether a model is equally likely to give a favourable prediction to the people... | ||
| Average Odds Difference | Metric | Architecture/model Agnostic Paradigm/supervised +1 | Any | Average Odds Difference summarises how differently a model treats two demographic groups, using two error rates. The... | ||
| Path-Specific Counterfactual Fairness Assessment | Metric | Architecture/model Agnostic Paradigm/supervised +1 | Tabular | Path-specific counterfactual fairness assessment tests whether a protected attribute, such as sex or race, influences a... | ||
| Embedding Bias Analysis | Testing | Architecture/neural Networks Architecture/neural Networks/transformer +3 | Text Image | Embedding bias analysis tests whether the numerical representations a model uses for words, images or documents link... | ||
| Few-Shot Fairness Evaluation | Testing | Architecture/neural Networks/transformer/llm Paradigm/generative +1 | Text | Few-shot fairness evaluation tests whether the worked examples placed in a prompt change how fairly a language model... | ||
| Conditional Demographic Disparity | Metric | Architecture/model Agnostic Requirements/black Box | Any | Conditional demographic disparity (CDD) measures whether members of a protected group make up a larger share of the... | ||
| Predictive Parity and Calibration Within Groups | Metric | Architecture/model Agnostic Paradigm/supervised +1 | Any | This test checks whether a model's output means the same for every group: among people who receive the same score or the... | ||
| Bias Benchmark for Question Answering | Testing | Architecture/neural Networks/transformer/llm Requirements/black Box | Text | The Bias Benchmark for Question Answering (BBQ) is a fixed set of about 58,500 multiple-choice reading questions that... | ||
| Rich Subgroup Fairness Auditing | Testing | Architecture/model Agnostic Paradigm/supervised +1 | Tabular | Rich subgroup fairness auditing checks whether a binary classifier treats any subgroup much better or much worse than... | ||
| Actionable Recourse Audit | Testing | Architecture/linear Models Architecture/linear Models/logistic +1 | Tabular | An actionable recourse audit checks whether the people a linear classifier refuses, for example for a loan, a job... |
Rows per page
Page 1 of 1