Description
Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.
Example Use Cases
Privacy
A genomics group plans to share a model trained on participants' genetic data. Before release, it runs shadow-model membership inference attacks and reports how many participants the attack picks out at a very low false-positive rate, to decide whether the model can be shared or needs stronger protection such as differential privacy.
A lender audits its credit-scoring model before sharing it with partner lenders. The audit compares attack results before and after regularisation, to decide whether the remaining leakage of which customers were in the training set is acceptable.
Security
A law firm fine-tunes a language model on confidential client contracts. A red team tests whether an outsider with query access can tell that a given contract was in the fine-tuning set, using member contracts and held-out contracts from the same period and source, to decide whether the model can be opened to wider use.
Limitations
- Attack success rates vary significantly depending on model architecture, training procedures, and data characteristics, making it difficult to establish universal thresholds for acceptable privacy.
- A failed attack proves little, because the result holds only for the attacks that were run. A published likelihood-ratio attack found members about ten times more often at low false-positive rates than earlier methods, so a model that resists weaker attacks may still leak, and no result amounts to a formal guarantee.
- Overall measures such as accuracy or area under the curve can look modest while a few records are identified with near certainty, and leakage tends to fall on unusual or out-of-distribution records. Reporting only an average therefore understates the risk to the individuals most exposed.
- Members and non-members must come from the same distribution. If non-members are drawn from a different period or source, as in many language-model benchmarks, the attack can learn that difference instead of membership and overstate the leakage.
- The strongest attacks are expensive. The likelihood-ratio attack trains hundreds of shadow models (256 for most datasets in its evaluation), which can be costly for large models, and a cheaper attack gives a weaker test.
Resources
Research Papers
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Whether Large Language models (LLMs) memorize their training data and what this means, from measuring privacy leakage to detecting copyright violations, has become a rapidly growing area of research. In the last few months, more than 10 new methods have been proposed to perform Membership Inference Attacks (MIAs) against LLMs. Contrary to traditional MIAs which rely on fixed-but randomized-records or models, these methods are mostly trained and tested on datasets collected post-hoc. Sets of members and non-members, used to evaluate the MIA, are constructed using informed guesses after the release of a model. This lack of randomization raises concerns of a distribution shift between members and non-members. In this work, we first extensively review the literature on MIAs against LLMs and show that, while most work focuses on sequence-level MIAs evaluated in post-hoc setups, a range of target models, motivations and units of interest are considered. We then quantify distribution shifts present in 6 datasets used in the literature, ranging from books to papers using a model-less bag of word classifier and compare the model-less results to the MIA. Our analysis shows that all of these datasets constructed post-hoc suffer from strong distribution shifts. These shifts invalidate the claims of LLMs memorizing strongly in real-world scenarios and, potentially, also the methodological contributions of the recent papers based on these datasets. Yet, all hope might not be lost. In the second part of this work, we introduce important considerations to properly evaluate MIAs against LLMs and discuss, in turn, potential ways forwards: randomized test splits, injections of randomized (unique) sequences, randomized finetuning, and several post-hoc control methods. While each option comes with its advantages and limitations, we believe they collectively provide solid grounds to guide the development of MIA methods and study LLM memorization. We conclude with an overview of recommended approaches to benchmark sequence-level and document-level MIAs against LLMs.
Membership Inference Attacks Against Machine Learning Models
We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on. We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.
Membership Inference Attacks From First Principles
A membership inference attack allows an adversary to query a trained machine learning model to predict whether or not a particular example was contained in the model’s training dataset. These attacks are currently evaluated using average-case “accuracy” metrics that fail to characterize whether the attack can confidently identify any members of the training set. We argue that attacks should instead be evaluated by computing their true-positive rate at low (e.g., ≤ 0.1%) false-positive rates, and find most prior attacks perform poorly when evaluated in this way. To address this we develop a Likelihood Ratio Attack (LiRA) that carefully combines multiple ideas from the literature. Our attack is 10× more powerful at low false-positive rates, and also strictly dominates prior attacks on existing metrics.