Causal Discrimination Testing
Description
Causal discrimination testing checks whether a system gives different outputs to otherwise identical inputs because of a protected attribute, by changing only that attribute, or a name, term or other proxy that signals it, and measuring how often or how much the output changes. Galhotra, Brun and Meliou (2017), who called the approach fairness testing, generate inputs from a description of the valid input fields, make copies of each that differ only in the protected attribute (for example, recorded sex), run them all through the system as a black box, and report the causal discrimination score: the share of inputs whose decision changes, sampled until it is known within a chosen margin of error and confidence level. For text, the same test substitutes identity terms or names in otherwise unchanged sentences, such as one religion or ethnicity for another, and reports the average change in the model's score across the substitutions, which Garg et al. (2019) call the counterfactual token fairness gap. Because every other part of the input is held fixed, the test needs no model of how the attribute influences other variables: it measures whether the system itself responds to the attribute, or to the chosen proxy. The output is a score per attribute and, if the tester logs them, the input pairs whose outputs differed.
Example Use Cases
Fairness
A bank's model validation team tests a credit-approval model before release. For a sample of last year's applications it changes only the applicant's recorded sex, and separately their marital status, re-scores each pair, and reports the share of decisions that flip with a confidence interval, against a release threshold agreed with the compliance function.
A trust-and-safety team checks a toxicity classifier used to hold comments for review by substituting identity terms such as 'gay', 'Muslim' or 'Black' across a set of non-toxic comments. It measures the average change in toxicity score for each term and flags any term whose substitution pushes benign comments over the moderation threshold.
An HR technology supplier tests its CV-ranking service before a client deploys it, by submitting the same CVs under first names and surnames associated with different ethnicities and genders and reporting how often each candidate's rank or shortlisting changes.
A hospital's AI assurance lead tests a symptom-triage assistant with clinical vignettes that differ only in the patient's stated ethnicity. Clinicians first mark which vignettes should be unaffected by ethnicity, and only those pairs count as failures when the urgency rating changes.
Limitations
- Holding every other input fixed measures only the system's direct response to the attribute or proxy. It does not follow through the effects the attribute has on other inputs, such as career breaks on an employment history. Garg et al. note that token substitution is more limited than counterfactual fairness defined on a causal model. Where those downstream effects matter, a clean result does not show that individuals were treated fairly.
- The test covers only what is perturbed. Unlisted identity terms, unperturbed proxies and multi-word references go undetected, as does discrimination that appears only when attributes change together: Galhotra et al. measured 11.2% for gender alone but 77.2% for gender, marital status and race together. Garg et al. found mitigation for listed terms generalised much less to held-out terms. A pass covers only the perturbations tried, and the list needs domain review.
- Generated inputs may be unrealistic. Themis draws inputs uniformly from the input schema, so the score covers combinations that never occur in practice. Testing real inputs or an existing test suite avoids that, but the score is then only as representative as that set. Galhotra et al. note that Themis computes no confidence level for it, and an interval from a random sample of real cases covers sampling error only.
- Not every substitution should leave the output unchanged. Garg et al. give 'That's so gay' versus 'That's so straight' as a pair a toxicity model may rightly score differently, and clinical risk can legitimately depend on sex. Without a judgement about which pairs must match, the test raises false alarms or, if such pairs are excluded wholesale, hides real failures.
- A model can pass and still produce very different outcome or error rates across groups, because the test asks only whether changing the attribute changes the output, not whether outcomes are equal. Galhotra et al. distinguish this causal discrimination from group discrimination, and Garg et al. note that a text classifier can satisfy counterfactual fairness while completely failing equality of odds, so it does not replace group metrics.
Resources
Research Papers
Fairness testing: testing software for discrimination
This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.
Counterfactual Fairness in Text Classification through Robustness
In this paper, we study counterfactual fairness in text classification, which asks the question: How would the prediction change if the sensitive attribute referenced in the example were different? Toxicity classifiers demonstrate a counterfactual fairness issue by predicting that "Some people are gay" is toxic while "Some people are straight" is nontoxic. We offer a metric, counterfactual token fairness (CTF), for measuring this particular form of fairness in text classifiers, and describe its relationship with group fairness. Further, we offer three approaches, blindness, counterfactual augmentation, and counterfactual logit pairing (CLP), for optimizing counterfactual token fairness during training, bridging the robustness and fairness literature. Empirically, we find that blindness and CLP address counterfactual token fairness. The methods do not harm classifier performance, and have varying tradeoffs with group fairness. These approaches, both for measurement and optimization, provide a new path forward for addressing fairness concerns in text classification.