Differential Privacy

Tutorial Planned

Description

Differential privacy puts a mathematical limit on what a published result reveals about any one person. The data holder adds random noise, drawn for example from a Laplace or Gaussian distribution, to each answer before release, such as a count, an average or a table of statistics, so the answer would look almost the same with or without any one person's data. The amount of noise depends on how much one person could change the answer (its sensitivity) and on a privacy budget called epsilon, which sets how far the two cases may differ (the Gaussian mechanism adds a second, small parameter, delta, for the chance of exceeding that limit). A smaller epsilon means more noise, a stronger guarantee and less accurate answers, and each release spends part of the budget. The guarantee covers whether a person's data took part, not conclusions about the wider population that the results legitimately show.

Example Use Cases

Privacy

A national statistics office adds calibrated noise to published census tables so that no single household's presence changes the results by more than a set amount. Statisticians use the epsilon setting to decide how much accuracy small areas must give up.

A health authority publishes weekly hospital admission counts by postcode district with Laplace noise added, so a rare diagnosis in a small district cannot be tied to one patient. Its privacy officer sets a yearly budget and decides how many weekly releases it can cover.

An analytics team lets internal analysts query customer purchase data only through an interface that adds noise and tracks each analyst's privacy budget. The data protection lead uses the budget log to decide when an analyst must stop querying.

Transparency

A model-evaluation team publishes noisy aggregate performance counts for declared groups. It documents the protected unit, sensitivity bound, noise mechanism and cumulative privacy budget so reviewers can interpret the release's guarantee.

Limitations

Noise lowers accuracy, and for a given epsilon the noise on a count does not shrink as the dataset grows, so small groups, rare categories and fine-grained tables suffer most. A strong guarantee (small epsilon) can make statistics for small subgroups unusable.
Setting the privacy budget (epsilon) requires expertise and careful consideration of the privacy-utility trade-off, with no universal guidelines for appropriate values across different applications.
Each release spends part of a fixed privacy budget, and the epsilons add up across releases. Once the budget is spent, no further analysis of the same data is possible without weakening the guarantee, so releases must be planned in advance.
The guarantee depends on correct sensitivity analysis: if a query's sensitivity is underestimated, or one person can contribute many records, too little noise is added and the stated epsilon no longer holds.
The guarantee concerns whether one person's data took part, not what the results show about the population. Conclusions that hold for everyone still apply to each individual, and protection weakens for groups, such as a family, by a factor equal to the group size.

Resources

Research Papers

The Algorithmic Foundations of Differential Privacy

Cynthia Dwork and Aaron Roth•Jan 1, 2014

The problem of privacy-preserving data analysis has a long history spanning multiple disciplines. As electronic data about individuals becomes increasingly detailed, and as technology enables ever more powerful collection and curation of these data, the need increases for a robust, meaningful, and mathematically rigorous definition of privacy, together with a computationally rich class of algorithms that satisfy this definition. Differential Privacy is such a definition. After motivating and discussing the meaning of differential privacy, the preponderance of this monograph is devoted to fundamental techniques for achieving differential privacy, and application of these techniques in creative combinations, using the query-release problem as an ongoing example. A key point is that, by rethinking the computational goal, one can often obtain far better results than would be achieved by methodically replacing each step of a non-private computation with a differentially private implementation. Despite some astonishingly powerful computational results, there are still fundamental limitations — not just on what can be achieved with differential privacy but on what can be achieved with any method that protects against a complete breakdown in privacy. Virtually all the algorithms discussed herein maintain differential privacy against adversaries of arbitrary computational power. Certain algorithms are computationally intensive, others are efficient. Computational complexity for the adversary and the algorithm are both discussed. We then turn from fundamentals to applications other than queryrelease, discussing differentially private methods for mechanism design and machine learning. The vast majority of the literature on differentially private algorithms considers a single, static, database that is subject to many analyses. Differential privacy in other models, including distributed databases and computations on data streams is discussed. Finally, we note that this work is meant as a thorough introduction to the problems and techniques of differential privacy, but is not intended to be an exhaustive survey — there is by now a vast amount of work in differential privacy, and we can cover only a small portion of it.

Calibrating Noise to Sensitivity in Private Data Analysis

Cynthia Dwork et al.•Jan 1, 2006

Software Packages

differential-privacy

Sep 4, 2019

Google's differential privacy libraries.

opacus

Dec 7, 2019

Training PyTorch models with differential privacy

Tutorials

Programming Differential Privacy

Jan 1, 2024

Documentations

Guidelines for evaluating differential privacy guarantees

Joseph P Near et al.•Mar 6, 2025

Related Techniques

Name	Description	Assurance Goals
Differentially Private Stochastic Gradient Descent	Differentially private stochastic gradient descent (DP-SGD) trains a neural network, or another model fitted by gradient descent, so that the finished model carries a formal differential privacy guarantee about the individual examples it was trained on. Differential privacy is more often applied by adding calibrated noise to the answer of a query or to a published statistic; DP-SGD instead changes the training algorithm itself. At each step it draws a random batch, computes the gradient (the direction in which an example pushes the model's parameters) separately for every example, scales down any gradient longer than a fixed clipping bound, adds Gaussian noise proportional to that bound to the sum, and updates the model with the noisy average, so that no single example can move the model by much and what influence remains is masked. A privacy accountant adds up the privacy cost of every step from the noise level, the sampling rate and the number of steps, and reports the total as epsilon and delta: epsilon limits how much more likely any trained model is with a given example in the training data than without it, and delta is a small probability that this limit fails. The output is a trained model with a stated (epsilon, delta) guarantee, which also covers anything later computed from the model, including its predictions.	Privacy
Differential Privacy Auditing	Differential privacy auditing tests whether a training procedure that claims a differential privacy guarantee at a stated budget (epsilon, where smaller means stronger privacy) loses more privacy in practice than the claim allows, by measuring a statistical lower bound on its true epsilon. The guarantee itself comes from a mathematical proof, and membership inference testing reports attack success on real records for any model; the audit instead controls which records are present and asks whether the code that was run honours the proof, so it can expose an implementation bug or accounting error but can never confirm the guarantee. The auditor builds two training sets that differ only by one or a few planted records, called canaries and often crafted to be easy to detect, trains the model many times on each, and applies a fixed test to every trained model to guess whether the canary was present. Confidence intervals for the test's true-positive and false-positive rates, such as Clopper-Pearson intervals for a proportion, are converted into a value of epsilon that the procedure must exceed at a stated confidence level, such as 99 per cent; a bound above the claimed epsilon shows that the claim is false for that implementation. Because hundreds of training runs are costly, a later design needs only one: many canaries are each included or left out at random, the test guesses which were included, and the share of correct guesses gives the bound, which is looser than the many-run design's.	Privacy
k-Anonymity	k-Anonymity is a property that a table of real records about people can be made to satisfy before it is released or shared, in which every record shares its values on the quasi-identifiers with at least k-1 other records, so anyone linking the table to another source on those columns always matches a group of at least k records, which is k people only when each person has one record. Quasi-identifiers are columns that are not names or ID numbers but could be found elsewhere and combined to single a person out, such as date of birth, sex and postcode, and because the data holder chooses them, the protection covers only those columns. To reach a chosen k, each quasi-identifier is coarsened along a hierarchy set in advance (exact age to five-year band, full postcode to district), and the values or records that would still sit in a group smaller than k are suppressed, with a search choosing the coarsening that loses least information. The output is the transformed table of real records made less specific, not artificial ones, together with the k it achieves and a measure of the detail lost. Unlike differential privacy, it gives no bound on what an observer can learn, because it blocks linking on the listed columns but does not stop someone learning a sensitive value that everyone in a group shares.	Privacy