API Usage Pattern Monitoring
Description
API usage pattern monitoring examines how each account uses a deployed model's API, to catch misuse that no single request reveals. The monitor keeps a running record per account and compares it with a baseline of normal use, raising an alert when the account departs from it. Published detectors for model theft show two ways to do this: one tests whether the distances between an account's successive queries follow the bell-shaped pattern that ordinary use produces, and another estimates how much of the model's behaviour the account's queries and responses would let someone rebuild. Providers also score how often an account triggers content filters or repeats jailbreak attempts, and send high scores for review. The output is a log of alerts and the actions taken, which gives a dated record of how the API was used compared with the uses it was approved for.
Example Use Cases
Security
A fraud-scoring API provider examines each client's query stream for runs of near-identical transactions altered in small steps, which suggest someone is probing for the score at which a transaction passes. Flagged accounts are rate-limited and sent to the security team for review.
A company selling traffic-sign recognition as a paid API tests whether the spacing between each customer's successive image queries follows the pattern ordinary use produces. Accounts whose queries look synthetic and widely spread, as in model-copying attacks, are flagged for investigation and possible suspension.
Safety
A health-technology company serving a paediatric triage model to many clinics monitors each clinic's query volumes and the ages in its requests. A clinic sending mostly adult patients, or a sudden change after a software update, is flagged so the company can contact it before unsafe advice is given.
A provider of a text-generation API scores each customer's usage by how often prompts and responses are flagged as harmful and whether jailbreak attempts recur. Customers with high scores go to review, which can lead to a warning or suspension.
Transparency
A bank's model risk team compares the monthly log of calls to its credit-scoring API, by internal team and purpose, with the uses approved at validation. The comparison is filed in the model's audit record, so reviewers can see whether the model is used as documented.
Limitations
- Anomalies are defined against a baseline of normal use, and ordinary users can be unusual. In PRADA's tests, legitimate queries made of consecutive frames from a sequence triggered false alarms at stricter settings, so the alert threshold has to be tuned to each model and use case.
- A patient attacker can stay inside normal limits by querying slowly, adding dummy queries or spreading queries across many accounts. PRADA's authors show that padding queries raises the attacker's cost three to ten times, and that multiple accounts defeat per-account analysis.
- Useful monitoring needs request content, or features of it, to be logged and sometimes read by reviewers. For sensitive applications that may be unlawful or unacceptable, and Microsoft notes that detection may be less accurate when human review is switched off for such customers.
- Detectors keep state for every account. PRADA's authors estimate a few gigabytes in total for a few hundred clients of an image model, and the cost rises with traffic and with the retention period set for audit evidence.
- The published extraction detectors were tested on small models, decision trees in one case and image classifiers in the other, against simulated attackers. Kesarwani et al. describe their warning as approximate, so an alert is a lead for investigation, not proof of theft.
Resources
Research Papers
Model Extraction Warning in MLaaS Paradigm
Machine learning models deployed on the cloud are susceptible to several security threats including extraction attacks. Adversaries may abuse a model's prediction API to steal the model thus compromising model confidentiality, privacy of training data, and revenue from future query payments. This work introduces a model extraction monitor that quantifies the extraction status of models by continually observing the API query and response streams of users. We present two novel strategies that measure either the information gain or the coverage of the feature space spanned by user queries to estimate the learning rate of individual and colluding adversaries. Both approaches have low computational overhead and can easily be offered as services to model owners to warn them against state of the art extraction attacks. We demonstrate empirical performance results of these approaches for decision tree and neural network models using open source datasets and BigML MLaaS platform.
PRADA: Protecting Against DNN Model Stealing Attacks
Machine learning (ML) applications are increasingly prevalent. Protecting the confidentiality of ML models becomes paramount for two reasons: (a) a model can be a business advantage to its owner, and (b) an adversary may use a stolen model to find transferable adversarial examples that can evade classification by the original model. Access to the model can be restricted to be only via well-defined prediction APIs. Nevertheless, prediction APIs still provide enough information to allow an adversary to mount model extraction attacks by sending repeated queries via the prediction API. In this paper, we describe new model extraction attacks using novel approaches for generating synthetic queries, and optimizing training hyperparameters. Our attacks outperform state-of-the-art model extraction in terms of transferability of both targeted and non-targeted adversarial examples (up to +29-44 percentage points, pp), and prediction accuracy (up to +46 pp) on two datasets. We provide take-aways on how to perform effective model extraction attacks. We then propose PRADA, the first step towards generic and effective detection of DNN model extraction attacks. It analyzes the distribution of consecutive API queries and raises an alarm when this distribution deviates from benign behavior. We show that PRADA can detect all prior model extraction attacks with no false positives.