Rich Subgroup Fairness Auditing

Description

Rich subgroup fairness auditing checks whether a binary classifier treats any subgroup much better or much worse than the population as a whole, where the subgroups are not a short list chosen in advance but every group that rules of a chosen kind over several protected attributes can describe, such as everyone for whom a weighted sum of age, sex and ethnicity indicators exceeds a threshold. Kearns, Neel, Roth and Wu (2018) proposed it to detect what they call fairness gerrymandering: a classifier can have equal false positive rates for men and for women, and for each ethnic group, yet very unequal rates for particular combinations, such as women of one ethnicity. For a chosen rate (the false positive or false negative rate, or in the paper also the rate of positive decisions), each subgroup is scored by the gap between its rate and the overall rate multiplied by the share of all records that are in the subgroup and count towards that rate (for the false positive rate, those with a negative true outcome), so a gap counts for more the larger the group, and the audit searches for the subgroup with the highest score. Exact search is computationally hard in the worst case, so it is posed as a weighted (cost-sensitive) classification problem over the protected attributes and solved approximately: the reference implementation fits two linear regressions that propose a candidate worst subgroup as a linear threshold rule, using only those attributes, the true outcomes and the classifier's decisions, not the model itself. The output is that rule with the subgroup's size, rate and weighted gap (called gamma), compared with a limit set beforehand; unlike reporting metrics for groups listed in advance, the audit finds subgroups nobody listed, and unlike multiaccuracy boosting, which checks a score's average error in each subgroup and then corrects the model, it compares a decision rate with the overall rate and changes nothing.

Example Use Cases

Fairness

A lender's model risk team validating a credit-risk model before release audits its decisions on a held-out sample of past applicants whose repayment is known, and records that these are only applicants the old policy approved. It searches subgroups over age, sex, ethnicity and disability status for the worst false positive rate gap (good payers flagged as high risk), compares it with a limit agreed with the risk committee, and sends the rule for any subgroup over it to the developers.

A hospital's clinical AI team has confirmed that a sepsis early-warning model misses cases at similar rates for men and women and for each recorded ethnic group. Before rollout it audits the false negative rate over subgroups combining age, sex and ethnicity and, because the search is approximate, also scores a grid of rules over two of the attributes by brute force, as the method's authors did, to check the search did not miss a larger gap.

A local authority's internal audit team re-audits a model that flags housing benefit claims for fraud review each quarter, using a random sample of claims reviewed whether or not they were flagged. It records the worst subgroup by false positive rate (legitimate claims flagged) over age, sex, ethnicity and disability, with its size and gap, and escalates to the system owner when the weighted gap exceeds the agreed limit.

Limitations

Finding the worst subgroup exactly is hard in the worst case, even for conjunctions or linear thresholds of attributes (Kearns et al. 2018, Theorem 3.3), so practical audits use a heuristic search. A subgroup it reports does have the measured gap on the audit data, but a clean result is only a lower bound: the authors note their auditor could miss a larger disparity and checked it by brute force on two attributes (Kearns et al. 2019). 'Nothing found' is a search result, not a proof.
Gaps are weighted by subgroup size, which the authors require for reliable estimates and which lets small groups pass with large gaps: a subgroup holding 2% of the records whose false positive rate is 20 percentage points above the overall rate scores at most 0.004, under the 0.005 limit used in the AIF360 demonstration. The audit cannot support a claim about a small named group; report that group's own rate, with an interval, instead.
Only subgroups that the rule class can describe, over the attributes designated as protected, are searched; the reference implementation uses linear threshold rules. The class has to be limited because, as the authors note, any imperfect classifier could be accused of unfairness to the set of people it misclassifies. A protected attribute that is not recorded or not designated is invisible, so the attributes used and their encoding must be stated with the finding.
Error-rate audits need the protected attributes and the true outcome for every audited record, which a deployed system may not supply, for example when outcomes are seen only for approved applications. Reliable estimates need more data the richer the subgroup class, the authors' experiments report in-sample results, and the tools give no confidence interval, so audit on held-out data and repeat on fresh data before relying on a worst-case figure.
It tests parity of one rate at a time for binary decisions, each subgroup's rate against the overall rate. A pass on false positives says nothing about false negatives, the AIF360 auditor implements only those two error rates, and other criteria, such as whether scores mean the same in every subgroup (calibration), need a different test. The rate must be chosen to match the fairness claim before the audit.

Resources

Research Papers

Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness

Michael Kearns et al.•Jul 3, 2018

The most prevalent notions of fairness in machine learning fix a small collection of pre-defined groups (such as race or gender), and then ask for approximate parity of some statistic of the classifier (such as false positive rate) across these groups. Constraints of this form are susceptible to fairness gerrymandering, in which a classifier is fair on each individual group, but badly violates the fairness constraint on structured subgroups, such as certain combinations of protected attribute values. We thus consider fairness across exponentially or infinitely many subgroups, defined by a structured class of functions over the protected attributes. We first prove that the problem of auditing subgroup fairness for both equality of false positive rates and statistical parity is computationally equivalent to the problem of weak agnostic learning — which means it is hard in the worst case, even for simple structured subclasses. However, it also suggests that common heuristics for learning can be applied to successfully solve the auditing problem in practice. We then derive an algorithm that provably converges in a polynomial number of steps to the best subgroup-fair distribution over classifiers, given access to an oracle which can solve the agnostic learning problem. The algorithm is based on a formulation of subgroup fairness as a zero-sum game between a Learner (the primal player) and an Auditor (the dual player). We implement a variant of this algorithm using heuristic oracles, and show that we can effectively both audit and learn fair classifiers on a real dataset.

Software Packages

AIF360

Aug 22, 2018

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Tutorials

AIF360 demo: GerryFair rich subgroup fairness (notebook)

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models. - Trusted-AI/AIF360

Documentations

aif360.metrics.BinaryLabelDatasetMetric (AIF360 documentation)

Related Techniques

Name	Description	Assurance Goals
Predictive Parity and Calibration Within Groups	This test checks whether a model's output means the same for every group: among people who receive the same score or the same decision, the share who turn out to have the outcome (or its average, for an outcome such as a cost or a count of illnesses) should not depend on their group, a requirement the fairness literature calls sufficiency. For a model that outputs a risk score, cases with known outcomes are grouped into score bins and, for each group, the observed outcome rate in each bin is plotted against the score, giving one calibration curve per group (a score is calibrated when, of the cases scored 0.3, about 30 per cent have the outcome); calibration within groups (Kleinberg, Mullainathan and Raghavan) holds when each group's curve matches the scores, and the weaker condition that the curves coincide holds when a given score corresponds to the same outcome rate in every group. For a yes/no decision the same comparison is made in the two cells of the decision: predictive parity (Chouldechova) holds when the positive predictive value, the share of people flagged positive who turn out positive, is equal across groups, and the negative predictive value can be compared in the same way for those not flagged. The two read-outs can disagree, because a score that means the same for every group can still give unequal positive predictive values at a threshold when the groups' scores are spread differently above it, so the test reports the read-out matching the output on which decisions are taken, with intervals and the gaps between groups. Where comparisons of error rates condition on the true outcome and ask whether people who turn out the same are treated the same, this test conditions on the model's output and asks whether people treated the same turn out the same; when the outcome is more common in one group, the two cannot both hold except in special cases.	Fairness
Demographic Parity Assessment	Demographic Parity Assessment checks whether a model gives a favourable outcome, such as a loan approval or an interview invitation, to each demographic group at the same rate. It compares those rates without adjusting for how many people in each group would qualify. Statistical Parity Difference is the gap between group rates (a signed difference between two named groups in AIF360, the highest rate minus the lowest across all groups in Fairlearn), and the Disparate Impact ratio divides the lowest rate by the highest. A ratio below 0.8 is a common warning sign, following the four-fifths rule used by US employment regulators. A gap shows that outcomes differ between groups but not why, and equal rates do not show that individuals were treated fairly.	Fairness
Causal Discrimination Testing	Causal discrimination testing checks whether a system gives different outputs to otherwise identical inputs because of a protected attribute, by changing only that attribute, or a name, term or other proxy that signals it, and measuring how often or how much the output changes. Galhotra, Brun and Meliou (2017), who called the approach fairness testing, generate inputs from a description of the valid input fields, make copies of each that differ only in the protected attribute (for example, recorded sex), run them all through the system as a black box, and report the causal discrimination score: the share of inputs whose decision changes, sampled until it is known within a chosen margin of error and confidence level. For text, the same test substitutes identity terms or names in otherwise unchanged sentences, such as one religion or ethnicity for another, and reports the average change in the model's score across the substitutions, which Garg et al. (2019) call the counterfactual token fairness gap. Because every other part of the input is held fixed, the test needs no model of how the attribute influences other variables: it measures whether the system itself responds to the attribute, or to the chosen proxy. The output is a score per attribute and, if the tester logs them, the input pairs whose outputs differed.	Fairness