Toxicity Detection
Description
Toxicity detection scores a model's outputs with a classifier trained to recognise abusive language, such as insults, threats, hate speech and profanity. Testers send the model a large set of prompts, including prompts that tend to provoke toxic replies, and score every response. Results are reported as the share of responses above a chosen score threshold, or as the worst score found across repeated attempts. Human reviewers check a sample of flagged and unflagged responses, because classifiers have known blind spots. The method measures offensive language, not every kind of harm, so dangerous instructions or manipulative advice can pass unflagged.
Example Use Cases
Safety
Before a chatbot goes live on a public-facing site used by children, its replies to a large set of provocative and adversarial prompts are scored with a toxicity classifier. The team decides whether it can launch by checking how many replies exceed an agreed toxicity threshold.
A bank tests its customer-complaints assistant with abusive and provocative messages, scoring every reply with a toxicity classifier, to decide whether it can go live without replying in kind or repeating offensive wording back to customers.
Before launching a mental health support chatbot, a clinical safety team scores its replies to simulated user messages with a toxicity classifier and has clinicians read a sample of flagged and unflagged replies, because subtle stigmatising wording can pass the classifier.
Limitations
- Classifiers can be biased. They lean on words such as slurs and profanity, so they tend to flag text that mentions minority identities or is written in dialects such as African American English, while missing polite but harmful text. Pass or fail results are only as fair as the classifier.
- Toxicity depends on context. The same phrase can be abuse, a quotation, reclaimed language or a discussion of abuse, and classifiers score text with little of that context. Expect false alarms in education, news and support settings, and misses where harm is implied rather than stated.
- Performance can drop outside the languages, dialects and slang a classifier was built for, and norms about what is offensive vary between communities and change over time. Multilingual or code-switched deployments need validation with local reviewers, and thresholds need re-checking.
- The pass mark is a choice. A low threshold blocks legitimate content and a high one lets harmful content through, and stakeholders often disagree about the balance. Report the threshold and the error rates it implies, or toxicity rates cannot be compared between systems.
- Subtle or coded harm, and harm that does not sound abusive, such as dangerous instructions or manipulative advice, can pass the classifier. Catching it needs human review of samples, which is slow, costly and distressing for reviewers, and may not scale to high-volume or real-time use.
Resources
Research Papers
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment.We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.We create and release RE-ALTOXICITYPROMPTS, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widelyused toxicity classifier.Using REALTOXICI-TYPROMPTS, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts.We empirically assess several controllable generation methods, and find that while data-or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration.To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et al., 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content.Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers
On the world wide web, toxic content detectors are a crucial line of defense against potentially hateful and offensive messages. As such, building highly effective classifiers that enable a safer internet is an important research area. Moreover, the web is a highly multilingual, cross-cultural community that develops its own lingo over time. As such, it is crucial to develop models that are effective across a diverse range of languages, usages, and styles. In this paper, we present the fundamentals behind the next version of the Perspective API from Google Jigsaw. At the heart of the approach is a single multilingual token-free Charformer model that is applicable across a range of languages, domains, and tasks. We demonstrate that by forgoing static vocabularies, we gain flexibility across a variety of settings. We additionally outline the techniques employed to make such a byte-level model efficient and feasible for productionization. Through extensive experiments on multilingual toxic comment classification benchmarks derived from real API traffic and evaluation on an array of code-switching, covert toxicity, emoji-based hate, human-readable obfuscation, distribution shift, and bias evaluation settings, we show that our proposed approach outperforms strong baselines. Finally, we present our findings from deploying this system in production.