Training Data Extraction Testing

Description

Training data extraction testing measures how much of its training text a generative language model reproduces word for word when it is sampled, without the tester supplying any training record to look for. It differs from membership inference testing, which starts from a candidate record and asks only whether that record was used in training, and from canary exposure testing, which needs secrets planted before training. The tester prompts the model with empty prompts or a few words taken from public text, generates a large volume of output (from tens of millions to billions of tokens) with the sampling settings the deployed system uses, and checks it against the training corpus: any span of at least a fixed length, such as 50 tokens, found verbatim in an index of the corpus, and not low-entropy filler such as repeated characters or lists of numbers, counts as extracted, and personal data is checked by tagging names and identifiers in the output and looking them up in the corpus. Where the corpus cannot be searched, the outputs are ranked by how much more probable the model finds them than a smaller reference model or a text-compression algorithm does, and the top-ranked candidates are checked by hand, for example by web search. The result is the share of generated text copied from training data (under the hand-checked route, the number confirmed among the candidates checked), the number of distinct passages extracted, and the passages themselves, each with a count of the training documents that contain it: a passage found in one or two documents is a privacy finding in a way that a licence or news story repeated thousands of times is not.

Example Use Cases

Privacy

A bank's privacy engineering team fine-tunes an open-weight language model on ten years of customer-service chat transcripts to draft replies for agents. Before deployment it generates 50 million tokens from public-text prompts with the production sampling settings, searches an index of the transcripts for 50-token matches, and looks up every account number, name and phone number tagged in the output. It runs the same test on the base model as a control.

A hospital research group that trained a clinical note-completion model on de-identified discharge summaries runs the test before sharing the weights with partner hospitals. Besides searching for verbatim passages of the summaries, it tags patient names, dates of birth and NHS numbers in the generated text and looks each up in the training set, to find identifiers the de-identification step missed and decide whether the summaries must be re-scrubbed and the model retrained.

An AI developer preparing an open-weight release compares two checkpoints, one trained on a deduplicated copy of the pre-training corpus. It generates text from each at budgets of 100 million and 1 billion tokens, counts distinct 50-token passages found verbatim in the corpus, and compares how the counts grow with the budget, not a single rate, before choosing which checkpoint to release and what to say about memorisation in the model card.

A government department running a citizen-facing chat assistant on a model it fine-tuned on casework correspondence repeats the test after every fine-tuning update. Because chat-tuned models can leak little under ordinary prompts yet emit training text once pushed off their chat behaviour, the test also uses prompts asking the assistant to repeat a single word many times, a pattern that made one production assistant reproduce its training data.

Limitations

The result is a lower bound set by the effort spent. The number of distinct passages extracted keeps growing as more text is generated, and one study found that a model which leaked more than another at a small budget leaked less in total. A clean result, or a comparison of models at one fixed budget, can therefore mislead; report the generation budget and compare models at several budgets.
Much memorised text comes out only after the right prompt. In one study, sampling from short public prompts recovered about a third of the passages a model was known to reproduce when given their true opening words, and a chat-tuned model leaked far less under ordinary prompts than when pushed into repeating a word. A low rate under ordinary prompts is weak evidence that the data cannot be extracted.
The verdict depends on what the output is checked against and on the match rule. A public proxy for an unavailable corpus undercounts (in one study a 9 TB proxy found under half the matches a manual web search found), and a minimum match length long enough to rule out coincidence misses short secrets such as phone or account numbers unless they are tagged and looked up separately. Paraphrased leaks and facts stitched together from several documents are not counted.
It is costly at a useful scale. One study generated a billion tokens per model, and its whole evaluation, including a 45 TB index built over a 9 TB corpus, took three weeks on a large cloud machine. The test also produces copies of any personal data the model leaks, which must be stored, reviewed and deleted under the same controls as the training data.
It shows what the model emits, not what an attacker could infer. It does not say whether a known person's record was used in training, and it gives no bound for data the test did not happen to elicit, so a clean result cannot stand in for a formal guarantee such as differential privacy.

Resources

Research Papers

Extracting Training Data from Large Language Models

Nicholas Carlini et al.•Jan 1, 2021

Software Packages

LM_Memorization (ftramer/LM_Memorization): training data extraction on GPT-2

Training data extraction on GPT-2. Contribute to ftramer/LM_Memorization development by creating an account on GitHub.

Tutorials

Analysing PII leakage (microsoft/analysing_pii_leakage)

The repository contains the code for analysing the leakage of personally identifiable (PII) information from the output of next word prediction language models. - microsoft/analysing_pii_leakage

Related Techniques

Name	Description	Assurance Goals
Canary Exposure Testing	Canary exposure testing measures how strongly a training procedure makes a generative sequence model, such as a language model, memorise rare sequences, by planting random secrets (canaries) in the training data and scoring how much the trained model favours them over values it never saw. Because the secrets are random and their values known, the result is calibrated in a way that tests on real records are not and can be compared across model versions, training settings and numbers of repetitions; unlike extraction or membership inference testing of a finished model, it requires control of the training run, and unlike differential privacy auditing, which also plants records, it gives a memorisation score rather than a statistical bound on epsilon. The tester chooses a format such as 'my account number is' followed by a blank, fills the blank with random values to make several canaries, inserts each into the training data a chosen number of times (for example once, ten times and a hundred times), and trains with the settings used in production. Each canary is then ranked by how probable the trained model finds it compared with every other possible filling of the blank, and its exposure is the base-2 logarithm of the number of possible fillings minus the base-2 logarithm of that rank: about 1 for a canary the model treats like any other value, rising to its maximum when the canary is the model's top choice, the level at which a search over the possible fillings can extract it outright. For large spaces of values the rank is estimated by fitting a distribution to the probabilities of a sample of alternatives, and results are usually plotted as exposure against number of insertions.	Privacy
Membership Inference Attack Testing	Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.	Privacy Security
Differential Privacy Auditing	Differential privacy auditing tests whether a training procedure that claims a differential privacy guarantee at a stated budget (epsilon, where smaller means stronger privacy) loses more privacy in practice than the claim allows, by measuring a statistical lower bound on its true epsilon. The guarantee itself comes from a mathematical proof, and membership inference testing reports attack success on real records for any model; the audit instead controls which records are present and asks whether the code that was run honours the proof, so it can expose an implementation bug or accounting error but can never confirm the guarantee. The auditor builds two training sets that differ only by one or a few planted records, called canaries and often crafted to be easy to detect, trains the model many times on each, and applies a fixed test to every trained model to guess whether the canary was present. Confidence intervals for the test's true-positive and false-positive rates, such as Clopper-Pearson intervals for a proportion, are converted into a value of epsilon that the procedure must exceed at a stated confidence level, such as 99 per cent; a bound above the claimed epsilon shows that the claim is false for that implementation. Because hundreds of training runs are costly, a later design needs only one: many canaries are each included or left out at random, the test guesses which were included, and the share of correct guesses gives the bound, which is looser than the many-run design's.	Privacy