Data Poisoning Detection

Description

Data poisoning detection looks for training examples that an attacker has planted to change how a model behaves. The best-studied case is the backdoor: a few examples carrying a hidden trigger, such as a sticker on a stop sign, are mislabelled so that the model misclassifies any input with the trigger while it performs normally on clean data. Detection first trains a model on the untrusted data, then examines how that model represents each training example internally (its activations in the last hidden layer), one class at a time. Activation clustering splits a class into two groups by those activations and flags the smaller group when the split is clear, and spectral signatures score each example by its position along the main direction of variation and flag the highest scores. The flagged examples are reviewed or removed and the model is retrained, which yields a list of suspected examples and a measure of how cleanly each class separates.

Example Use Cases

Safety

A team building a traffic-sign classifier for a driver-assistance system trains a trial model on images gathered from several suppliers and runs activation clustering on each class. A small separate group of stop-sign images labelled as speed limits, all carrying the same sticker, is flagged and removed before the final training run.

A platform trains a harmful-content classifier on labels supplied by crowd workers. Analysts scan each class for a small separate group of examples sharing a trigger phrase and labelled as harmless, which would let that phrase slip past the filter, and remove them before retraining.

Security

A research platform builds a chest X-ray classifier from images contributed by many partner hospitals. Before any partner's images enter the training set, it trains a trial model and scores each class for the small, separable group of examples a planted backdoor leaves, because one contributor's data could be compromised.

An insurer trains a damage-assessment classifier on claims photos labelled by an outside vendor. Before relying on the labels, it scores each class for a small, separate group of photos carrying the same mark and labelled undamaged, which a compromised vendor account could have planted so that damaged vehicles with the mark pass unchecked. Flagged photos go back to the vendor for explanation.

Limitations

An attacker who knows the detector can adapt. Both methods rely on poisoned examples standing out in the model's learned representation, so an attack that keeps them close to genuine examples can slip through. Chen et al. found no obvious way round activation clustering within their threat model, but say further work is needed.
False alarms can remove legitimate but unusual examples, reducing data quality and diversity. Chen et al. suggest a silhouette-score cut-off of about 0.10 to 0.15 based on three datasets; another data type or model may need different values, and a wrong value either removes good data or leaves poison in.
Both methods assume poisoned examples are a minority that separates from clean ones in the learned representation, and Tran et al.'s needs an upper bound on how many there are. They were designed and tested on backdoors in image and text classifiers, so attacks that only lower overall accuracy lie outside what the original papers show.
Both methods first train a model on the untrusted data, read its internal activations, then retrain after removing flagged examples. They need the full training data and white-box access to the model, and cost at least two training runs, so they cannot be applied to a model someone else trained when the data or weights are unavailable.
Real poison is unlabelled, so a detector's accuracy can only be measured on poison the tester injects, and a clean result shows that this detector found nothing, not that the data is clean. Telling poison from naturally unusual data is hard.

Resources

Research Papers

Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

Bryant Chen et al.•Nov 9, 2018

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

Spectral Signatures in Backdoor Attacks

Brandon Tran, Jerry Li, and Aleksander Madry•Nov 1, 2018

A recent line of work has uncovered a new form of data poisoning: so-called \emph{backdoor} attacks. These attacks are particularly dangerous because they do not affect a network's behavior on typical, benign data. Rather, the network only deviates from its expected output when triggered by a perturbation planted by an adversary. In this paper, we identify a new property of all known backdoor attacks, which we call \emph{spectral signatures}. This property allows us to utilize tools from robust statistics to thwart the attacks. We demonstrate the efficacy of these signatures in detecting and removing poisoned examples on real image sets and state of the art neural network architectures. We believe that understanding spectral signatures is a crucial first step towards designing ML systems secure against such backdoor attacks

Software Packages

Adversarial Robustness Toolbox (Trusted-AI/adversarial-robustness-toolbox)

Oct 8, 2026

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Tutorials

ART notebook: poisoning defense with activation clustering

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams - Trusted-AI/adversarial-robustness-toolbox

Documentations

art.defences.detector.poison — Adversarial Robustness Toolbox ...

Adversarial-robustness-toolbox Developers

Related Techniques

Name	Description	Assurance Goals
Malicious Model-File Scanning	Malicious model-file scanning checks a downloaded or received model file for code that would run when the file is loaded, without loading it. Several common ways of saving (serialising) models can carry such code, above all Python's pickle format, the default for PyTorch checkpoints and the usual format for scikit-learn models, whose files can instruct the loader to import and call any Python function, so that loading an untrusted file can hand an attacker control of the machine; since version 2.6 PyTorch's own loader by default allows only a small set of known-safe functions, but some libraries switch that off to load their models. The named method is ModelScan, an open-source scanner from Protect AI, which chooses a check by file extension and reads the file as data: for pickle-based files it lists the functions the file would import and compares them with a graded list of unsafe ones, such as the os, subprocess and socket modules and Python's built-in eval and exec, and for Keras and TensorFlow files it flags Lambda layers, which hold arbitrary Python, and operations that read or write files. It reports each finding with a severity from critical to low and exits with a non-zero code when it finds any, so a pipeline can stop the file before anything loads it. The evidence is a record that no listed unsafe call was found in the files scanned; unlike data-poisoning detection it examines the saved file rather than the training data, and it shows neither that the file is safe, nor how the model behaves, nor who produced the file.	Security
Membership Inference Attack Testing	Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.	Privacy Security
Adversarial Training Evaluation	Adversarial training evaluation tests whether a model trained on adversarial examples (inputs altered slightly to cause errors) is really more robust, or has only learned to resist the attacks used in training. The evaluator first states a threat model, meaning the kind and size of change allowed and what the attacker knows. The model is then attacked with a varied set of strong methods, including ones not used in training, ones adapted to this particular defence, and ones that do not rely on the model's gradients, which exposes models that merely hide them. The main output is robust accuracy, the share of inputs still classified correctly under attack, plotted against the allowed size of the change and reported next to accuracy on unaltered inputs. Because attacks can only find weaknesses, the result is an upper bound on robustness, not a guarantee.	Security