Synthetic Data Evaluation

Description

Synthetic data evaluation checks whether an artificial dataset made to stand in for sensitive real records is safe to share and still useful, and reports the trade-off between the two. Privacy is tested by simulating an attacker who holds the synthetic data and some facts about a person, and measuring how accurately the attacker can use the data to infer that person's sensitive values. Fidelity is tested by comparing the synthetic and real data column by column and in pairs of columns. Another check trains a classifier to tell the two apart, and a classifier that cannot beat chance suggests they are statistically alike. Utility is tested by running the analyses or training the models the data is meant for on the synthetic data and comparing the results with those from the real data.

Example Use Cases

Privacy

A hospital research office wants to share synthetic outpatient records with university partners. Before release it simulates an attacker who knows each patient's age band, sex and postcode district and tries to infer their diagnosis from the synthetic table, and it withholds the data if the attack does better than guessing from overall diagnosis rates.

Reliability

A national statistics office gives researchers synthetic microdata for exploratory analysis. After the researchers settle on their models, the office runs the same models on the real data inside a secure environment and compares the estimates, to tell the researchers whether their exploratory conclusions can be trusted.

A bank's fraud team generates synthetic card transactions to share with an external vendor. It compares how often each fraud type appears and how amount, merchant type and time of day relate in the real and synthetic data, then checks that a model trained on the synthetic data still catches rare fraud in held-out real transactions.

Limitations

Stronger privacy protection usually costs data quality, so the three scores pull against each other. There is no neutral threshold: where to set the balance depends on how sensitive the data is and what it will be used for, and the evaluation reports the trade-off without settling it.
Attack-based privacy tests only probe the attacks and attacker knowledge that were tried. A dataset that resists them may still leak to an attacker with more auxiliary information or a better method, so a good result is not proof of safety.
Fidelity and utility scores depend on which metrics, models and analyses are chosen. A synthetic dataset can match the real data on the columns and column pairs checked yet differ in ways that matter for a particular analysis, and general and analysis-specific utility scores can disagree.
Scores summarise the whole table, so rare events and small subgroups can be lost without moving the totals much. A fraud pattern or a minority group that the generator has smoothed away may show up only if the evaluation is designed to look for it.
The tests need statistical, privacy-engineering and domain expertise to set up and read. A classifier-based fidelity check can be misled by over-fitting or under-fitting the classifier, and weak settings can make poor synthetic data look good.

Resources

Research Papers

General and Specific Utility Measures for Synthetic Data

Joshua Snoke et al.•Mar 7, 2018

Data holders can produce synthetic versions of data sets when concerns about potential disclosure restrict the availability of the original records. The paper is concerned with methods to judge whether such synthetic data have a distribution that is comparable with that of the original data: what we term general utility. We consider how general utility compares with specific utility: the similarity of results of analyses from the synthetic data and the original data. We adapt a previous general measure of data utility, the propensity score mean-squared error pMSE, to the specific case of synthetic data and derive its distribution for the case when the correct synthesis model is used to create the synthetic data. Our asymptotic results are confirmed by a simulation study. We also consider two specific utility measures, confidence interval overlap and standardized difference in summary statistics, which we compare with the general utility results. We present two contrasting examples of data syntheses: one illustrating synthetic data that is evaluated as being useful by both general and specific measures and the second where neither is the case. For the second case we show how the general utility measures can identify the deficiencies of the synthetic data and suggest how this can inform possible improvements to the synthesis method.

Software Packages

sdv-dev/SDMetrics: Metrics to evaluate quality and efficacy of synthetic datasets

Oct 1, 2026

Metrics to evaluate quality and efficacy of synthetic datasets.

Tutorials

Quality Report | SDMetrics

Sep 28, 2026

Documentations

Welcome to TAPAS's documentation! — tapas 0.1 documentation

Tapas-privacy Developers•Jan 1, 2023

SDMetrics | Synthetic Data Metrics

Jan 28, 2026

Related Techniques

Name	Description	Assurance Goals
Membership Inference Attack Testing	Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.	Privacy Security
Attacker-Model Re-identification Risk Measurement	Attacker-model re-identification risk measurement estimates, for a table of real records about people that is about to be released or shared, the probability that an attacker who links it with other information could correctly match a record to the person it describes. This record follows the attacker models of El Emam and Dankar (prosecutor and journalist) and of Dankar and El Emam (marketer), as implemented in the ARX tool: the assessor lists the quasi-identifiers (columns such as age, sex, postcode and admission date that an attacker could know or find), groups the records that share the same values on them, and names the attacker being guarded against. For a prosecutor, who knows the target is in the table, a record's risk is one over the size of its group in the table; for a journalist, who does not know this and links to a register of the wider population, it is one over the size of the matching group in that population, estimated with a statistical model when the population counts are not known; for a marketer, who tries to match as many records as possible, the measure is the expected share of records matched correctly. The assessor reports the highest record risk, the average risk and the share of records above a threshold set for the release, which decides whether the table can go out as it is or needs more generalisation or suppression first. It measures how easily records can be matched to people under the stated assumptions, unlike k-anonymity, which transforms the data, and it does not measure what can be learned about someone from values their whole group shares.	Privacy
Differential Privacy Auditing	Differential privacy auditing tests whether a training procedure that claims a differential privacy guarantee at a stated budget (epsilon, where smaller means stronger privacy) loses more privacy in practice than the claim allows, by measuring a statistical lower bound on its true epsilon. The guarantee itself comes from a mathematical proof, and membership inference testing reports attack success on real records for any model; the audit instead controls which records are present and asks whether the code that was run honours the proof, so it can expose an implementation bug or accounting error but can never confirm the guarantee. The auditor builds two training sets that differ only by one or a few planted records, called canaries and often crafted to be easy to detect, trains the model many times on each, and applies a fixed test to every trained model to guess whether the canary was present. Confidence intervals for the test's true-positive and false-positive rates, such as Clopper-Pearson intervals for a proportion, are converted into a value of epsilon that the procedure must exceed at a stated confidence level, such as 99 per cent; a bound above the claimed epsilon shows that the claim is false for that implementation. Because hundreds of training runs are costly, a later design needs only one: many canaries are each included or left out at random, the test guesses which were included, and the share of correct guesses gives the bound, which is looser than the many-run design's.	Privacy