Canary Exposure Testing

Description

Canary exposure testing measures how strongly a training procedure makes a generative sequence model, such as a language model, memorise rare sequences, by planting random secrets (canaries) in the training data and scoring how much the trained model favours them over values it never saw. Because the secrets are random and their values known, the result is calibrated in a way that tests on real records are not and can be compared across model versions, training settings and numbers of repetitions; unlike extraction or membership inference testing of a finished model, it requires control of the training run, and unlike differential privacy auditing, which also plants records, it gives a memorisation score rather than a statistical bound on epsilon. The tester chooses a format such as 'my account number is' followed by a blank, fills the blank with random values to make several canaries, inserts each into the training data a chosen number of times (for example once, ten times and a hundred times), and trains with the settings used in production. Each canary is then ranked by how probable the trained model finds it compared with every other possible filling of the blank, and its exposure is the base-2 logarithm of the number of possible fillings minus the base-2 logarithm of that rank: about 1 for a canary the model treats like any other value, rising to its maximum when the canary is the model's top choice, the level at which a search over the possible fillings can extract it outright. For large spaces of values the rank is estimated by fitting a distribution to the probabilities of a sample of alternatives, and results are usually plotted as exposure against number of insertions.

Example Use Cases

Privacy

An email provider training a reply-suggestion model on users' messages inserts seven-word canaries with random middle words between 1 and 10,000 times each into a training run with the production settings, plots exposure against the number of insertions, and uses the curve to limit how often any rare phrase from a single user may appear in the training data before the model is offered to all users.

A bank's machine learning team choosing among learning-rate schedules and model sizes for an autocomplete model trained on transaction descriptions trains each candidate with the same random account-number canaries at the same insertion counts, and picks the configuration with the lowest exposure among those within a set margin of the best validation accuracy.

A health-records software vendor uses the test as a regression check: each time its clinical-text model is retrained on new notes, the same canaries are inserted at fixed frequencies, and the build fails if the exposure of a canary inserted only once rises above a threshold agreed with the vendor's information governance team.

Limitations

It needs control of the training run. Canaries must be in the data before training, so the test cannot be applied to a model someone else trained or has already released, and it must use the production settings (optimiser, epochs, model size, hyperparameters), because each of these changes how much is memorised; a test model trained differently says little about the deployed one.
It measures memorisation of the planted secrets, not of the real data. Canaries are random and unlike the rest of the data by design, while real personal data that is repeated, structured or similar to other text can be memorised differently, and the choice of format and of random values changes the result. Low exposure for the canaries does not show that no real record can be extracted.
Exposure is not a privacy guarantee. Its scale depends on the number of possible fillings, so values compare only across canaries with the same number of fillings, and it is not an epsilon: in the original study, models trained with differential privacy at budgets too weak to mean anything (epsilon up to 10^9) still showed negligible exposure, so low exposure cannot stand in for a formal bound.
It needs the model's probability for arbitrary sequences, so a system that returns only its single best output needs a different measure. When the space of possible fillings is too large to score in full, exposure is estimated by fitting a distribution to a sample of alternatives; the estimate depends on that fit holding in the far tail and can exceed the true maximum.

Resources

Research Papers

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Nicholas Carlini et al.•Jan 1, 2019

Software Packages

TensorFlow Privacy (tensorflow/privacy)

Library for training machine learning models with privacy for training data - tensorflow/privacy

Tutorials

Assess privacy risks with TensorFlow Privacy Secret Sharer Attack (example notebook)

Library for training machine learning models with privacy for training data - tensorflow/privacy

Documentations

Secret Sharer Attack (TensorFlow Privacy, privacy_tests/secret_sharer)

Library for training machine learning models with privacy for training data - tensorflow/privacy

Related Techniques

Name	Description	Assurance Goals
Training Data Extraction Testing	Training data extraction testing measures how much of its training text a generative language model reproduces word for word when it is sampled, without the tester supplying any training record to look for. It differs from membership inference testing, which starts from a candidate record and asks only whether that record was used in training, and from canary exposure testing, which needs secrets planted before training. The tester prompts the model with empty prompts or a few words taken from public text, generates a large volume of output (from tens of millions to billions of tokens) with the sampling settings the deployed system uses, and checks it against the training corpus: any span of at least a fixed length, such as 50 tokens, found verbatim in an index of the corpus, and not low-entropy filler such as repeated characters or lists of numbers, counts as extracted, and personal data is checked by tagging names and identifiers in the output and looking them up in the corpus. Where the corpus cannot be searched, the outputs are ranked by how much more probable the model finds them than a smaller reference model or a text-compression algorithm does, and the top-ranked candidates are checked by hand, for example by web search. The result is the share of generated text copied from training data (under the hand-checked route, the number confirmed among the candidates checked), the number of distinct passages extracted, and the passages themselves, each with a count of the training documents that contain it: a passage found in one or two documents is a privacy finding in a way that a licence or news story repeated thousands of times is not.	Privacy
Differential Privacy Auditing	Differential privacy auditing tests whether a training procedure that claims a differential privacy guarantee at a stated budget (epsilon, where smaller means stronger privacy) loses more privacy in practice than the claim allows, by measuring a statistical lower bound on its true epsilon. The guarantee itself comes from a mathematical proof, and membership inference testing reports attack success on real records for any model; the audit instead controls which records are present and asks whether the code that was run honours the proof, so it can expose an implementation bug or accounting error but can never confirm the guarantee. The auditor builds two training sets that differ only by one or a few planted records, called canaries and often crafted to be easy to detect, trains the model many times on each, and applies a fixed test to every trained model to guess whether the canary was present. Confidence intervals for the test's true-positive and false-positive rates, such as Clopper-Pearson intervals for a proportion, are converted into a value of epsilon that the procedure must exceed at a stated confidence level, such as 99 per cent; a bound above the claimed epsilon shows that the claim is false for that implementation. Because hundreds of training runs are costly, a later design needs only one: many canaries are each included or left out at random, the test guesses which were included, and the share of correct guesses gives the bound, which is looser than the many-run design's.	Privacy
Membership Inference Attack Testing	Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.	Privacy Security

Canary Exposure Testing

Description

Example Use Cases

Privacy

Limitations

Resources

Research Papers

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Software Packages

TensorFlow Privacy (tensorflow/privacy)

Tutorials

Assess privacy risks with TensorFlow Privacy Secret Sharer Attack (example notebook)

Documentations

Secret Sharer Attack (TensorFlow Privacy, privacy_tests/secret_sharer)

Tags

Privacy Dimensions

Other Categories