Differentially Private Stochastic Gradient Descent

Description

Differentially private stochastic gradient descent (DP-SGD) trains a neural network, or another model fitted by gradient descent, so that the finished model carries a formal differential privacy guarantee about the individual examples it was trained on. Differential privacy is more often applied by adding calibrated noise to the answer of a query or to a published statistic; DP-SGD instead changes the training algorithm itself. At each step it draws a random batch, computes the gradient (the direction in which an example pushes the model's parameters) separately for every example, scales down any gradient longer than a fixed clipping bound, adds Gaussian noise proportional to that bound to the sum, and updates the model with the noisy average, so that no single example can move the model by much and what influence remains is masked. A privacy accountant adds up the privacy cost of every step from the noise level, the sampling rate and the number of steps, and reports the total as epsilon and delta: epsilon limits how much more likely any trained model is with a given example in the training data than without it, and delta is a small probability that this limit fails. The output is a trained model with a stated (epsilon, delta) guarantee, which also covers anything later computed from the model, including its predictions.

Example Use Cases

Limitations

• Clipping and noise reduce accuracy, and the reduction is not shared evenly: classes and groups that are rare in the training data lose the most, so a private model can widen an existing accuracy gap between groups. In the original experiments the drop at epsilon 8 was about 1 percentage point on handwritten digits but about 7 on a harder ten-class image task. Accuracy should be reported per group, and pretraining on public data or more training data is often needed to keep the cost acceptable.
• The stated epsilon holds only if the code matches the accountant's assumptions. The usual accountants assume each batch is drawn by including every example independently at random, but many pipelines shuffle the data and cut it into fixed batches; an audit of models trained that way measured privacy loss up to four times the epsilon they reported. A claim should name the sampling method and accountant, and differential privacy auditing can test the figure empirically.
• The guarantee protects one training example, not one person. If a patient has ten admissions or a customer hundreds of messages in the training set, their protection is much weaker than epsilon suggests, because the bound loosens with the number of examples they contribute. Capping each person's contribution, or making each person a single example, gives a guarantee at the level of the person, at some cost in data and accuracy.
• No agreed value of epsilon makes a model private, and large values are common: the method's original headline results used epsilon 8, at which a training outcome may be about 3,000 times more likely with a given example than without it, and the Opacus image tutorial targets 50. Hyperparameter searches run on the private data also leak information unless their cost is added to the budget. A claim should state epsilon, delta, the unit of privacy and what the budget covers.
• Computing a separate gradient for every example costs memory and time; in the original experiments one training epoch went from about 40 to about 180 seconds. Layers that combine examples within a batch, such as batch normalisation, break the per-example bound and must be replaced, for example with group normalisation, so an existing architecture may need changing and re-tuning before it can be trained privately.

Resources

Related Techniques

Tags