Adversarial Training Evaluation
Description
Adversarial training evaluation tests whether a model trained on adversarial examples (inputs altered slightly to cause errors) is really more robust, or has only learned to resist the attacks used in training. The evaluator first states a threat model, meaning the kind and size of change allowed and what the attacker knows. The model is then attacked with a varied set of strong methods, including ones not used in training, ones adapted to this particular defence, and ones that do not rely on the model's gradients, which exposes models that merely hide them. The main output is robust accuracy, the share of inputs still classified correctly under attack, plotted against the allowed size of the change and reported next to accuracy on unaltered inputs. Because attacks can only find weaknesses, the result is an upper bound on robustness, not a guarantee.
Example Use Cases
Security
A building-access provider trains its face-recognition model with adversarial examples from a single standard attack. Before relying on it, the security team attacks the model with methods not used in training, including a gradient-free attack, and reports robust accuracy at the perturbation size in its threat model, to check the gain is not overfitting to the training attack.
A mail provider adversarially trains its spam filter against text-perturbation attacks. The evaluation measures how much attacker-altered spam still gets through when attacks not used in training are applied, and how the false-positive rate on ordinary mail has changed, to decide whether the clean-accuracy cost of the hardened filter is acceptable.
A hospital adversarially trains a tumour-detection classifier for scans. The evaluation attacks it with a strong attack ensemble at several perturbation sizes and compares accuracy on routine scans before and after hardening, to decide whether the hardened model is safe to deploy in the screening workflow.
A driver-assistance supplier trains its road-sign classifier against changes of up to a fixed size per pixel. The evaluation also attacks it with a different kind of change (a limit on total distortion across the image) and with rotations and shifts, to see whether the robustness holds outside the threat model used in training.
Limitations
- The result is an upper bound on robustness, not a guarantee. Robust accuracy is measured against the attacks that were run, so a model that survives them can still be broken by a stronger attack, and an evaluation can never prove that no attack exists.
- Findings hold only for the threat model tested. Robustness to changes of one kind and size says little about other kinds or larger sizes, so each result must be reported with the perturbation type, size and attacker knowledge it assumed.
- Weak or poorly tuned attacks overstate robustness. Default settings, a single attack or the attack used in training can leave a model looking robust, and gradient masking can make gradient-based attacks fail. In one large study, a standard attack ensemble lowered the robust accuracy reported for 13 of over 50 published models by more than 10%.
- A thorough evaluation is expensive and needs expertise. Strong attacks run many iterations and restarts, gradient-free and hard-label attacks need far more queries, and attacks often have to be adapted to the defence by someone who understands how it works.
- Attack methods keep improving, so a result is dated. The evaluation should be repeated when stronger attacks appear and should include attacks not used in training or tuning, which means keeping a current set of attack methods.
Resources
Research Papers
On Evaluating Adversarial Robustness
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making it difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation are improper tuning of hyperparameters of the attacks, gradient obfuscation or masking. In this paper we first propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness. We apply our ensemble to over 50 models from papers published at recent top machine learning and computer vision venues. In all except one of the cases we achieve lower robust test accuracy than reported in these papers, often by more than $10\%$, identifying several broken defenses.