Intriguing properties of neural networks

Inside the paper

The problem

Neural networks performed well on image recognition, but it was difficult to understand what their hidden units represented or how their predictions behaved near familiar images.

The idea

The authors examine both questions directly: compare individual units with combinations of units, then search for small input changes that produce a chosen wrong prediction.

How it works

• For a held-out set, collect images that strongly activate one hidden unit, and compare them with images activating a random direction in the same representation.
• To seek a misleading image, choose a target label and optimize a perturbation while keeping the resulting pixels within their allowed range.
• The optimization trades off the size of the change against loss for the target label, using a constrained numerical procedure to approximate the solution.
• The authors test the resulting images on other networks, including models trained with different settings or disjoint subsets of the training data.

What it showed

Both individual units and random directions produced groups of visually related images. The optimized changes fooled the source models, and a substantial fraction also fooled other models in the experiments.

Read it for

Start with Section 3 and Figures 1–4 for the representation comparison, then read Section 4.1 and Figure 5 for the perturbation experiment. The later mathematical bounds can be skimmed on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.