Deep Residual Learning for Image Recognition

Inside the paper

The problem

Deeper image classifiers were promising, but simply adding layers could make training error worse, even when a shallower network could in principle be embedded in the deeper one.

The idea

The authors reframe a group of layers as learning a change to its input, with a shortcut carrying the input alongside that change. This gives the optimizer a direct route to an identity mapping.

How it works

• A residual block sends its input through several learned layers and adds their output to the input carried by a shortcut.
• When dimensions change, the shortcut can use zero padding or a learned projection so the two paths can be added.
• The authors construct otherwise comparable plain and residual networks, train them from scratch, and compare their training and validation errors.
• For deeper models they also use a bottleneck block that changes channel dimensions around a spatial convolution.

What it showed

On ImageNet and CIFAR-10, deeper plain networks showed higher training error, while their residual counterparts trained more successfully and gained accuracy with depth. On the ImageNet test set, the residual ensemble made a top-5 error on 3.57% of images.

Read it for

Start with Figure 1 and Sections 3.1–3.2 for the training problem and the block, then compare the curves in Figure 4. You can skim the implementation details in Section 3.4 on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.