Model Extraction Defence Testing

Description

Model extraction defence testing checks whether a model offered through a query interface, such as a prediction API, can be copied by someone who can only send it inputs and read its outputs. The tester plays the attacker, spends a fixed query budget on the protected interface, then uses the answers to train a surrogate model (a copy trained on the answers) or to solve for the model's parameters, and compares the copy with the original. Fidelity is usually measured as the share of inputs on which the copy agrees with the original, together with how many queries that took. The same attack is run with and without each defence, such as limiting queries, rounding or removing confidence scores, or adding noise to outputs, to show how much each defence raises the attacker's cost. The tester also measures the effect on legitimate users, for example accuracy or response time.

Example Use Cases

Security

Testing protections for a proprietary fraud detection API to ensure competitors cannot recreate the model's decision boundaries through systematic querying, by simulating extraction attacks using query budgets, active learning strategies, and substitute model training.

Evaluating whether rate limiting and output obfuscation for an automated essay grading API prevent competitors from extracting the scoring model through systematic submission of probe essays designed to reverse-engineer grading criteria.

A transport authority sells access to a traffic-prediction API and rounds its forecasts to deter copying. It runs extraction attacks against rounded and unrounded versions to see how much rounding makes a surrogate lose fidelity, then checks that the rounded forecasts remain accurate enough for urban planners.

Limitations

A defence is tested only against the attacks the tester thinks to run, and attackers adapt. A 2024 attack recovered part of production language models through log-probability and logit-bias options that designers had treated as harmless, so passing a test is not a guarantee of protection.
Defences that perturb or hide outputs can reduce utility for legitimate users, and the acceptable loss depends on the application. The test has to measure legitimate accuracy and latency as well as extraction fidelity, or it will reward defences that quietly break the service.
It is hard to tell legitimate high-volume use from extraction, so query limits and detection can block valid users. Published stateful detection defences have themselves been bypassed, so a detector that passes a test may still fail against an adaptive attacker.
Results depend on what the simulated attacker is assumed to know, such as the model type, the training-data distribution, and whether the API returns confidence scores or accepts partial inputs. Testing against a weak assumed attacker overstates protection.
There is no agreed threshold for how much copying fidelity is acceptable, and a fidelity figure depends on the query budget and the measure used. Results are therefore hard to compare across models and defences, and the pass mark is a judgement.

Resources

Research Papers

Stealing Machine Learning Models via Prediction APIs

Florian Tramèr et al.•Oct 3, 2016

Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis. The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial feature vectors as inputs and include confidence values with predictions. Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning. We further show that the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks. Our results highlight the need for careful ML model deployment and new model extraction countermeasures.

Software Packages

Adversarial Robustness Toolbox (Trusted-AI/adversarial-robustness-toolbox)

Oct 8, 2026

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Tutorials

Model stealing demo notebook — ART notebooks

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams - Trusted-AI/adversarial-robustness-toolbox

Documentations

art.attacks.extraction — Adversarial Robustness Toolbox documentation

Related Techniques

Name	Description	Assurance Goals
Adversarial Training Evaluation	Adversarial training evaluation tests whether a model trained on adversarial examples (inputs altered slightly to cause errors) is really more robust, or has only learned to resist the attacks used in training. The evaluator first states a threat model, meaning the kind and size of change allowed and what the attacker knows. The model is then attacked with a varied set of strong methods, including ones not used in training, ones adapted to this particular defence, and ones that do not rely on the model's gradients, which exposes models that merely hide them. The main output is robust accuracy, the share of inputs still classified correctly under attack, plotted against the allowed size of the change and reported next to accuracy on unaltered inputs. Because attacks can only find weaknesses, the result is an upper bound on robustness, not a guarantee.	Security
Membership Inference Attack Testing	Membership inference attack testing checks whether someone with access to a trained model can tell if a particular record was in its training data. The tester plays the attacker, using the model's confidence scores, its prediction patterns or its loss on a record, often with shadow models, which are copies trained on similar data whose members are known. The attack is scored on records known to be members and records known to be non-members, which the tester must keep apart. The key result is the share of true members the attack identifies when it wrongly flags only a tiny share of non-members (for example 0.1%), because overall accuracy can hide an attack that identifies a few records with near certainty. The results guide the choice of mitigations, such as differential privacy or regularisation.	Privacy Security
Jailbreak Resistance Testing	Jailbreak resistance testing checks whether a language model or chatbot can be talked out of its safety rules. Testers write or collect prompts that try to elicit content the system is meant to refuse, such as role-playing a character without restrictions, framing the request as hypothetical, hiding it in an encoding such as Base64, or combining several of these. Each response is judged as a refusal, a breach or unclear, by a reviewer or an automated check. The result is the rate at which each attack family succeeds, with the example prompts that worked. This shows where the safeguards fail, but only for the attacks tried.	Safety Security