Calibration with Equality of Opportunity

Description

Calibration with equality of opportunity is a post-processing step that adjusts a classifier's scores so that two groups face the same error rate, in a score-based form of the equal opportunity criterion, while the scores stay calibrated. Calibrated means that, within each group, about 70% of the people given a score of 0.7 really have the outcome. The error rate is computed from the scores rather than from yes-or-no decisions, and for equality of opportunity it is how far, on average, the scores of people who truly have the outcome fall short of 1. Starting from scores already calibrated within each group, the step replaces a randomly chosen share of the scores in the group with the lower error rate by that group's overall rate of the outcome, which makes those predictions less informative until the two error rates match. Calibration is kept at a cost in accuracy for that group, and matching this score-based rate does not guarantee equal true positive rates once a decision threshold is applied.

Example Use Cases

Fairness

A hospital team finds that its follow-up screening model gives lower risk scores to women who go on to develop the condition than to men who do. Using a held-out validation set with recorded sex, it adjusts the scores so that this error is equal across the two groups, then measures the accuracy lost by the group whose scores were replaced, to decide whether the model can be released.

Reliability

A university uses a model to offer extra support to students at risk of dropping out, and wants a score of 0.3 to mean about a 30% dropout rate in every fee-status group. After equalising the error rate across groups, planners plot calibration by group on held-out data to confirm the scores still mean what they say before using them to allocate support places.

Limitations

The target can be infeasible. If even replacing every score in the lower-error group with that group's base rate leaves its error below the other group's, the two rates cannot be matched while staying calibrated. This happens when the other group's classifier is close to uninformative.
Equality is reached by making predictions worse for the group with the lower error rate, not by improving them for the other group. A random share of that group's scores is replaced by its base rate, and the loss grows with the gap between the groups' base rates.
Calibration can coexist with only one error constraint. Matching the score-based false negative rate leaves false positive rates free to diverge, so one group can face more false alarms, and equalising both while staying calibrated is impossible except in trivial cases.
Because the replaced predictions are chosen at random, two similar people in the same group can receive different outcomes. Those affected may reasonably object that a consequential decision was made by withholding information, which makes the method hard to justify in high-stakes settings.
It needs each person's group attribute when scores are adjusted, labelled held-out data to estimate how many scores to replace, and starting scores already calibrated within each group, which it preserves but cannot create. Where the attribute cannot lawfully be used, or the base model is miscalibrated, the method does not apply.

Resources

Research Papers

On Fairness and Calibration

Geoff Pleiss et al.•Sep 6, 2017

The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

Equality of Opportunity in Supervised Learning

Moritz Hardt, Eric Price, and Nathan Srebro•Oct 7, 2016

We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. In line with other studies, our notion is oblivious: it depends only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation of individualfeatures. We study the inherent limits of defining and identifying biases based on such oblivious measures, outlining what can and cannot be inferred from different oblivious tests. We illustrate our notion using a case study of FICO credit scores.

Software Packages

equalized_odds_and_calibration

Nov 3, 2017

Code and data for the experiments in "On Fairness and Calibration"

AIF360

Aug 22, 2018

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Tutorials

AIF360 demo: Calibrated Equalized Odds Postprocessing (notebook)

Documentations

aif360.algorithms.postprocessing.CalibratedEqOddsPostprocessing — aif360 documentation

Related Techniques

Name	Description	Assurance Goals
Threshold Optimiser	Threshold Optimiser is a post-processing method, introduced by Hardt, Price and Srebro (2016), that adjusts a trained classifier's decisions to meet a group fairness criterion without retraining it, most often equalised odds (equal true positive and false positive rates in every group) or equal opportunity (equal true positive rates only). It uses a labelled validation set that records each person's group, and traces each group's ROC curve, which shows the true and false positive rates the model's score gives at every possible cut-off. It then chooses a separate cut-off for each group that keeps accuracy, or another chosen objective, as high as the criterion allows. Equalised odds usually cannot be met with one cut-off per group, so each group's decision is randomised between two cut-offs with fixed probabilities. The output is a rule that needs each person's group at prediction time, and if the model gives only yes/no predictions, a small linear optimisation sets the probability of flipping each prediction instead.	Fairness
Reject Option Classification	Reject Option Classification is a post-processing method, proposed by Kamiran and colleagues (2012), that changes some of a trained classifier's decisions to reduce unfair outcomes between groups, without retraining it. It defines a 'critical region' around the decision boundary, where the predicted probability is close to 50 per cent and the model is least sure. Inside that region it gives the favourable outcome to people from the disadvantaged group and the unfavourable outcome to people from the advantaged group, whatever the model predicted, and decisions outside it are left alone. The width of the region is searched on a labelled validation set, so that a chosen fairness measure, such as statistical parity difference or equal opportunity difference, falls within a set bound while accuracy stays as high as possible. The method needs each person's group at prediction time.	Fairness
Predictive Parity and Calibration Within Groups	This test checks whether a model's output means the same for every group: among people who receive the same score or the same decision, the share who turn out to have the outcome (or its average, for an outcome such as a cost or a count of illnesses) should not depend on their group, a requirement the fairness literature calls sufficiency. For a model that outputs a risk score, cases with known outcomes are grouped into score bins and, for each group, the observed outcome rate in each bin is plotted against the score, giving one calibration curve per group (a score is calibrated when, of the cases scored 0.3, about 30 per cent have the outcome); calibration within groups (Kleinberg, Mullainathan and Raghavan) holds when each group's curve matches the scores, and the weaker condition that the curves coincide holds when a given score corresponds to the same outcome rate in every group. For a yes/no decision the same comparison is made in the two cells of the decision: predictive parity (Chouldechova) holds when the positive predictive value, the share of people flagged positive who turn out positive, is equal across groups, and the negative predictive value can be compared in the same way for those not flagged. The two read-outs can disagree, because a score that means the same for every group can still give unequal positive predictive values at a threshold when the groups' scores are spread differently above it, so the test reports the read-out matching the output on which decisions are taken, with intervals and the gaps between groups. Where comparisons of error rates condition on the true outcome and ask whether people who turn out the same are treated the same, this test conditions on the model's output and asks whether people treated the same turn out the same; when the outcome is more common in one group, the two cannot both hold except in special cases.	Fairness