Average Odds Difference

Description

Average Odds Difference summarises how differently a model treats two demographic groups, using two error rates. The true positive rate is the share of people whose true outcome is positive whom the model also predicts as positive. The false positive rate is the share of people whose true outcome is negative whom the model wrongly predicts as positive. The score is the mean of the gap in each rate between the groups, usually the unprivileged group minus the privileged group, so 0 means the two gaps are absent or cancel out. Hardt, Price and Srebro (2016) call the stricter requirement that both gaps are zero equalised odds, and a score near 0 does not guarantee it.

Example Use Cases

Fairness

A court-services analytics team audits a pre-trial risk tool on past cases with known outcomes. It reports the average odds difference between racial groups together with the two gaps it averages: how often defendants who later reoffended were rated high risk, and how often those who did not were wrongly rated high risk. The findings go to the oversight board that decides whether the tool stays in use.

A recruitment firm audits its CV-screening tool on past applications whose suitability hiring managers later confirmed. For men and women it compares how often suitable applicants were advanced and how often unsuitable applicants were advanced, and it tracks the average of the two gaps each quarter to decide whether the tool needs retraining.

A hospital compares a sepsis early-warning model across patient ethnic groups before rollout. It checks the share of true sepsis cases flagged and the share of non-sepsis patients falsely flagged, since missed cases and false alarms both carry costs, and uses the average gap to rank candidate models.

Limitations

Averaging effect can mask important disparities when false positive and true positive rate differences compensate for each other, potentially hiding significant bias in one direction.
The rates are only as trustworthy as the outcome labels, which can differ in quality between groups. Where outcomes are recorded only for people previously approved or hired, such as repayment for granted loans, the gaps describe a selected subset rather than the whole applicant pool. The sensitive attribute must also be available and lawful to use.
When groups differ in how many members truly have the positive outcome, equal error rates generally cannot be achieved together with scores that mean the same thing in every group (Hardt et al. discuss this result of Kleinberg et al.). Pushing the metric towards 0 can therefore leave a risk score less well calibrated for one group, a trade-off the metric does not weigh.
A score of 0 does not show that the model is fair. The measure uses only predictions, true outcomes and group membership, and Hardt et al. show that situations with very different fairness readings can give identical numbers, so any result needs scrutiny of how the outcome was produced.
Closing the gaps can mean degrading the better-served group instead of improving the other. Hardt et al. show that a predictor meeting equalised odds is only as good as the weakest group's performance, so reaching a score of 0 this way can cost accuracy for everyone.

Resources

Research Papers

Equality of Opportunity in Supervised Learning

Moritz Hardt, Eric Price, and Nathan Srebro•Oct 7, 2016

We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. In line with other studies, our notion is oblivious: it depends only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation of individualfeatures. We study the inherent limits of defining and identifying biases based on such oblivious measures, outlining what can and cannot be inferred from different oblivious tests. We illustrate our notion using a case study of FICO credit scores.

AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias

R. K. E. Bellamy et al.•Jul 1, 2019

Fairness is an increasingly important concern as machine learning models are used to support decision making in high-stakes applications such as mortgage lending, hiring, and prison sentencing. This article introduces a new open-source Python toolkit for algorithmic fairness, AI Fairness 360 (AIF360), released under an Apache v2.0 license (https://github.com/ibm/aif360). The main objectives of this toolkit are to help facilitate the transition of fairness research algorithms for use in an industrial setting and to provide a common framework for fairness researchers to share and evaluate algorithms. The package includes a comprehensive set of fairness metrics for datasets and models, explanations for these metrics, and algorithms to mitigate bias in datasets and models. It also includes an interactive Web experience that provides a gentle introduction to the concepts and capabilities for line-of-business users, researchers, and developers to extend the toolkit with their new algorithms and improvements and to use it for performance benchmarking. A built-in testing infrastructure maintains code quality.

Software Packages

fairlearn

May 15, 2018

A Python package to assess and improve fairness of machine learning models.

AIF360

Aug 22, 2018

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Tutorials

AIF360 tutorial: Medical Expenditure (notebook)

Documentations

AIF360: Average Odds Difference Documentation

Aif360 Developers•Jan 1, 2018

Related Techniques

Name	Description	Assurance Goals
Equal Opportunity Difference	Equal Opportunity Difference checks whether a model is equally likely to give a favourable prediction to the people whose true outcome is favourable, such as loan applicants who would repay, in each demographic group. For each group it calculates the true positive rate: of the people whose true outcome is positive, the share the model also predicts as positive. The score is the gap between two groups' rates, usually the unprivileged group's minus the privileged group's, and 0 means the rates match. Some libraries report the largest gap across all groups instead. The metric follows the equality of opportunity criterion of Hardt, Price and Srebro (2016), and it says nothing about people whose true outcome is negative, so a score of 0 does not show that the model is fair overall.	Fairness
Demographic Parity Assessment	Demographic Parity Assessment checks whether a model gives a favourable outcome, such as a loan approval or an interview invitation, to each demographic group at the same rate. It compares those rates without adjusting for how many people in each group would qualify. Statistical Parity Difference is the gap between group rates (a signed difference between two named groups in AIF360, the highest rate minus the lowest across all groups in Fairlearn), and the Disparate Impact ratio divides the lowest rate by the highest. A ratio below 0.8 is a common warning sign, following the four-fifths rule used by US employment regulators. A gap shows that outcomes differ between groups but not why, and equal rates do not show that individuals were treated fairly.	Fairness
Predictive Parity and Calibration Within Groups	This test checks whether a model's output means the same for every group: among people who receive the same score or the same decision, the share who turn out to have the outcome (or its average, for an outcome such as a cost or a count of illnesses) should not depend on their group, a requirement the fairness literature calls sufficiency. For a model that outputs a risk score, cases with known outcomes are grouped into score bins and, for each group, the observed outcome rate in each bin is plotted against the score, giving one calibration curve per group (a score is calibrated when, of the cases scored 0.3, about 30 per cent have the outcome); calibration within groups (Kleinberg, Mullainathan and Raghavan) holds when each group's curve matches the scores, and the weaker condition that the curves coincide holds when a given score corresponds to the same outcome rate in every group. For a yes/no decision the same comparison is made in the two cells of the decision: predictive parity (Chouldechova) holds when the positive predictive value, the share of people flagged positive who turn out positive, is equal across groups, and the negative predictive value can be compared in the same way for those not flagged. The two read-outs can disagree, because a score that means the same for every group can still give unequal positive predictive values at a threshold when the groups' scores are spread differently above it, so the test reports the read-out matching the output on which decisions are taken, with intervals and the gaps between groups. Where comparisons of error rates condition on the true outcome and ask whether people who turn out the same are treated the same, this test conditions on the model's output and asks whether people treated the same turn out the same; when the outcome is more common in one group, the two cannot both hold except in special cases.	Fairness