Disaggregated Evaluation

Description

Disaggregated evaluation assesses and reports a system's performance separately for each group of people a claim covers, and for combinations of groups (intersectional groups, such as darker-skinned women), so that a good overall score cannot hide a group for whom the system works badly. Following Barocas et al. (2021), the evaluator first fixes the design choices that change what the evaluation can show: which groups and combinations to report; how other factors that affect performance, such as lighting or recording conditions, are handled (held constant, varied deliberately, or left as they occur); which metrics to use, for a classifier often false positive and false negative rates reported separately, because each can harm different people; and whether the evaluation is exploratory, or confirmatory, meaning that it tests hypotheses stated in advance on data built to hold enough cases in every group. Each metric is then computed on a labelled evaluation set for every group and combination, together with the number of cases behind it and an interval showing how far the estimate could move through sampling alone, obtained for example by resampling the evaluation set. Groups are compared by absolute and by relative gaps, which can tell different stories: false negative rates of 1 and 10 per cent differ by nine points but tenfold. The output is a table or chart of per-group and per-combination results with their case counts and intervals, and a record of the design choices needed to interpret them; it is this design, more than any one metric or interval method, that decides which comparisons the evaluation can support and how conclusive each is.

Example Use Cases

Fairness

Before a hospital trust adopts a supplier's skin-lesion classifier, its clinical safety team reports sensitivity and specificity on locally labelled images for each skin-tone group, sex and age band, and for skin tone crossed with sex. Cells holding fewer melanoma cases than the evaluation protocol requires are marked as too small to estimate rather than reported, and the trust collects more images for them before deciding whether the tool may be used for all patients.

A consumer organisation evaluating the speech-to-text services behind automated phone lines has speakers from several regional and ethnic groups, matched on age and gender, read identical scripted passages, and reports word error rate per group with intervals. Because everyone says the same words, a gap can be attributed to how each service recognises speech rather than to what was said, the evidence the organisation needs for a complaint to the providers.

A bank's model risk team runs a confirmatory evaluation of its fraud model each quarter. The groups (age band, sex and their combinations), the metrics (false positive rate, which drives blocked legitimate payments, and false negative rate), the sample and the largest acceptable gap are fixed in a protocol before the data are drawn, and the model stays in service only if every interval for a gap lies inside that margin.

An exam board piloting an automated essay-scoring model runs an exploratory evaluation of its agreement with human markers, split by first language, school type and their combinations. It finds one weak cell, candidates with English as an additional language at small schools, too small to be conclusive, and commissions a confirmatory study with a sample built to fill that cell before deciding whether the scores can count towards grades.

Limitations

Each attribute added multiplies the number of cells and shrinks each one, so combinations of attributes soon hold too few cases for a stable estimate, and where a cell holds only a few errors, intervals from resampling come out too narrow. Comparing many cells also throws up some large gaps by chance. A confirmatory claim needs enough cases in every cell, which Barocas et al. note existing datasets rarely provide, so the data available limit which comparisons can be tested.
A gap between groups shows that performance differs, not why. Another factor that affects performance may differ between the groups, as with the time between paired images that NIST suggested could explain a higher false negative rate for Asian people in face recognition. Holding such a factor constant hides gaps at its other values, varying it needs the factor labelled for every case, and strongly correlated factors are hard to separate, or impossible when they always occur together.
The result is only as good as two sets of labels. Where ground-truth outcomes are wrong, or recorded differently for different groups, and the extent of the error cannot be identified, Barocas et al. note the results are meaningless however careful the design. Group membership is often ascribed by an institution or inferred from appearance, adding errors and possible harm to those labelled, and datasets may hold few of some groups (darker-skinned women were 4.4 per cent of one benchmark).
It measures differences in performance and nothing else. Equal error rates say nothing about whether outputs stereotype or demean people, or whether favourable decisions are shared in proportion, and narrowing a gap under one metric can open a gap under another. A clean result supports a claim about the performance gaps measured, for the groups and conditions tested, and no wider fairness claim.
Results on a laboratory dataset may not carry over to the system in use, where lighting, recording conditions and operators' decisions differ, and Barocas et al. note that a component's performance alone may say little about the whole system around it. Evaluating in the setting of use captures this but is costly, and is often impossible for an outside evaluator, who usually sees the system only as a black box.

Resources

Research Papers

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas et al.•Jul 21, 2021

Software Packages

fairlearn

May 15, 2018

A Python package to assess and improve fairness of machine learning models.

Tutorials

Performing a Fairness Assessment (Fairlearn user guide)

Documentations

fairlearn.metrics.MetricFrame (Fairlearn API reference)

Related Techniques

Name	Description	Assurance Goals
Predictive Parity and Calibration Within Groups	This test checks whether a model's output means the same for every group: among people who receive the same score or the same decision, the share who turn out to have the outcome (or its average, for an outcome such as a cost or a count of illnesses) should not depend on their group, a requirement the fairness literature calls sufficiency. For a model that outputs a risk score, cases with known outcomes are grouped into score bins and, for each group, the observed outcome rate in each bin is plotted against the score, giving one calibration curve per group (a score is calibrated when, of the cases scored 0.3, about 30 per cent have the outcome); calibration within groups (Kleinberg, Mullainathan and Raghavan) holds when each group's curve matches the scores, and the weaker condition that the curves coincide holds when a given score corresponds to the same outcome rate in every group. For a yes/no decision the same comparison is made in the two cells of the decision: predictive parity (Chouldechova) holds when the positive predictive value, the share of people flagged positive who turn out positive, is equal across groups, and the negative predictive value can be compared in the same way for those not flagged. The two read-outs can disagree, because a score that means the same for every group can still give unequal positive predictive values at a threshold when the groups' scores are spread differently above it, so the test reports the read-out matching the output on which decisions are taken, with intervals and the gaps between groups. Where comparisons of error rates condition on the true outcome and ask whether people who turn out the same are treated the same, this test conditions on the model's output and asks whether people treated the same turn out the same; when the outcome is more common in one group, the two cannot both hold except in special cases.	Fairness
Rich Subgroup Fairness Auditing	Rich subgroup fairness auditing checks whether a binary classifier treats any subgroup much better or much worse than the population as a whole, where the subgroups are not a short list chosen in advance but every group that rules of a chosen kind over several protected attributes can describe, such as everyone for whom a weighted sum of age, sex and ethnicity indicators exceeds a threshold. Kearns, Neel, Roth and Wu (2018) proposed it to detect what they call fairness gerrymandering: a classifier can have equal false positive rates for men and for women, and for each ethnic group, yet very unequal rates for particular combinations, such as women of one ethnicity. For a chosen rate (the false positive or false negative rate, or in the paper also the rate of positive decisions), each subgroup is scored by the gap between its rate and the overall rate multiplied by the share of all records that are in the subgroup and count towards that rate (for the false positive rate, those with a negative true outcome), so a gap counts for more the larger the group, and the audit searches for the subgroup with the highest score. Exact search is computationally hard in the worst case, so it is posed as a weighted (cost-sensitive) classification problem over the protected attributes and solved approximately: the reference implementation fits two linear regressions that propose a candidate worst subgroup as a linear threshold rule, using only those attributes, the true outcomes and the classifier's decisions, not the model itself. The output is that rule with the subgroup's size, rate and weighted gap (called gamma), compared with a limit set beforehand; unlike reporting metrics for groups listed in advance, the audit finds subgroups nobody listed, and unlike multiaccuracy boosting, which checks a score's average error in each subgroup and then corrects the model, it compares a decision rate with the overall rate and changes nothing.	Fairness
Demographic Parity Assessment	Demographic Parity Assessment checks whether a model gives a favourable outcome, such as a loan approval or an interview invitation, to each demographic group at the same rate. It compares those rates without adjusting for how many people in each group would qualify. Statistical Parity Difference is the gap between group rates (a signed difference between two named groups in AIF360, the highest rate minus the lowest across all groups in Fairlearn), and the Disparate Impact ratio divides the lowest rate by the highest. A ratio below 0.8 is a common warning sign, following the four-fifths rule used by US employment regulators. A gap shows that outcomes differ between groups but not why, and equal rates do not show that individuals were treated fairly.	Fairness