Explaining and Harnessing Adversarial Examples

Inside the paper

The problem

Image classifiers could be fooled by small, deliberately chosen input changes, but the reason such examples existed and worked across models was contested.

The idea

The authors argue that approximately linear behavior in high dimensions can add many small input changes into a large change in a model’s response. They also use such changes during training.

How it works

• For a given example, calculate the gradient of the training loss with respect to its input, holding the model parameters fixed.
• Change each input coordinate in the direction indicated by the sign of its gradient, with a fixed bound on the size of each change.
• Add these altered examples to the training objective alongside ordinary examples, so the network learns from inputs chosen to increase its loss.
• Compare this procedure with alternatives including random input noise, and examine examples made for one model on other models.

What it showed

On MNIST, adversarial training reduced error on examples generated by the tested gradient-sign attack, though errors remained. The paper also found that a generative model and an ensemble were vulnerable in its experiments.

Read it for

Read Sections 3 and 4 for the linear argument and attack, then Section 6 for the training experiment. Figure 1 gives a visual example; Sections 7–9 explore alternative explanations and can follow later.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.