Disparate Impact Remover

Description

Disparate Impact Remover is a pre-processing method that edits the numerical features of a dataset so that they reveal less about a protected attribute such as race or sex. For each feature it compares how the values are distributed within each protected group, then moves every value towards a shared distribution while keeping its rank within its own group, so people near the top of their group stay near the top. The user sets a repair level from 0 (no change) to 1 (each feature is distributed identically in every group). The aim is to avoid disparate impact, where one group's selection rate falls below 80 per cent of the most favoured group's rate. Stronger repair narrows that gap but costs predictive accuracy.

Example Use Cases

Fairness

A lender's data science team finds that income, length of employment and credit history length in its loan dataset differ systematically between applicants under and over 25. It repairs those three features at a level of 0.8, trains its scoring model on the repaired data, and checks the approval-rate ratio between the age groups against the 80 per cent threshold to decide whether a stronger repair is needed.

A fire service reviewing a promotion process finds that written and oral exam scores differ between ethnic groups, so a 70 per cent pass mark selects far fewer candidates from one group. It repairs both scores while keeping each candidate's rank within their own group, then checks whether the repaired scores still separate stronger candidates from weaker ones.

Limitations

Repair costs predictive accuracy, and the cost varies sharply by dataset. In the original experiments a full repair cut a measure of classifier utility on the German credit data from about 72 per cent to about 50 per cent, close to chance, but the Adult income data lost only about two points. The repair level has to be chosen and justified.
It repairs numerical features only, and the AIF360 implementation takes one protected attribute at a time. Categorical features such as job title or postcode are left unchanged and can still carry the protected attribute, and repairing several attributes together means the user must define the joint groups.
Each feature is repaired separately, so every feature can look identical across groups while combinations of features still reveal the protected attribute. A model trained on several repaired features may therefore still show disparate impact, which is why the original paper pairs the repair with a test that tries to predict the protected attribute from all the features together.
The repair depends on each person's protected attribute and on the group distributions seen in the training data. AIF360 notes that new data can be transformed the same way only if those conditional distributions match, and each new record needs its protected attribute, which makes the method awkward for a live scoring service.
The target is the US 80 per cent rule, a comparison of group selection rates, and the paper notes that other legal frameworks may need a different formulation. The rule says nothing about how individuals are treated, and the paper points out that disparate impact is not itself unlawful where a business necessity can be shown.

Resources

Research Papers

Certifying and Removing Disparate Impact

Michael Feldman et al.•Aug 10, 2015

What does it mean for an algorithm to be biased? In U.S. law, unintentional bias is encoded via disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. This legal determination hinges on a definition of a protected class (ethnicity, gender) and an explicit description of the process. When computers are involved, determining disparate impact (and hence bias) is harder. It might not be possible to disclose the process. In addition, even if the process is open, it might be hard to elucidate in a legal setting how the algorithm makes its decisions. Instead of requiring access to the process, we propose making inferences based on the data it uses. We present four contributions. First, we link disparate impact to a measure of classification accuracy that while known, has received relatively little attention. Second, we propose a test for disparate impact based on how well the protected class can be predicted from the other attributes. Third, we describe methods by which data might be made unbiased. Finally, we present empirical evidence supporting the effectiveness of our test for disparate impact and our approach for both masking bias and preserving relevant information in the data. Interestingly, our approach resembles some actual selection practices that have recently received legal scrutiny.

Software Packages

holisticai

Jan 26, 2023

This is an open-source tool to assess and improve the trustworthiness of AI systems.

AIF360

Aug 22, 2018

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Tutorials

AIF360 demo: Disparate Impact Remover (notebook)

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models. - Trusted-AI/AIF360

Documentations

Disparate Impact Remover — holisticai documentation

Holisticai Developers•Jan 1, 2023

aif360.algorithms.preprocessing.DisparateImpactRemover — aif360 ...

Aif360 Developers•Jan 1, 2018

Related Techniques

Name	Description	Assurance Goals
Attribute Removal (Fairness Through Unawareness)	Attribute removal, also called fairness through unawareness, trains and runs a model without the protected attributes, such as race, gender or age, among its inputs. This meets the unawareness criterion, under which an algorithm counts as fair so long as it does not explicitly use a protected attribute, and it rules out direct discrimination on those attributes. It does not make outcomes fair. Other features can act as proxies, meaning they correlate with a removed attribute and let the model reconstruct it, for example a postcode standing in for ethnicity. For that reason the technique is best seen as the simplest baseline, to be combined with methods that look at outcomes.	Fairness
Relabelling	Relabelling, also called data massaging, is a pre-processing method proposed by Kamiran and Calders. It changes some of the outcome labels in a training set so that a protected group and everyone else have the same rate of positive outcomes, such as being hired or approved. A ranking model first scores each example by how likely it is to be positive. The method then promotes the highest-scoring negative examples from the disadvantaged group to positive and demotes the lowest-scoring positive examples from the favoured group to negative, in equal numbers. Because the examples closest to the decision boundary change first and the numbers are equal, the overall share of positive labels stays the same and the number of changes is as small as it can be.	Fairness
Preferential Sampling	Preferential Sampling is a pre-processing method, proposed by Kamiran and Calders, that rebalances a training set so that a protected group and everyone else have the same rate of positive outcomes, such as being hired or approved. A ranking model first scores every example by how likely it is to have a positive label. The method then compares the size of each group-and-outcome combination with the size it would have if group and outcome were unrelated. It adds copies of positive examples from the disadvantaged group and negative examples from the favoured group, and removes negative examples from the disadvantaged group and positive examples from the favoured group, always starting with those closest to the decision boundary. Labels are never changed, so a standard classifier can learn from the rebalanced set.	Fairness