ImageNet Classification with Deep Convolutional Neural Networks

Inside the paper

The problem

Recognising objects in photos at the scale of ImageNet's 1.2 million labelled images and 1,000 classes needed a model with a lot of capacity, but large neural networks were slow to train and quick to overfit.

The idea

Train a large convolutional network directly on raw pixels, made practical by fast GPU code, and keep it from memorising the training set with data augmentation and dropout.

How it works

• Five convolutional layers, some followed by max-pooling, then three fully connected layers ending in a 1,000-way softmax; about 60 million parameters in all.
• ReLU units instead of tanh or sigmoid, so the network trains several times faster.
• The network is split across two GPUs that communicate only at certain layers, because one GPU's memory could not hold it.
• Random crops, horizontal flips and colour changes enlarge the training set; dropout, applied to the first two dense layers, stops units from leaning on each other.

What it showed

A deep network learned from pixels alone could win ImageNet by a wide margin: its top-5 test error in the 2012 competition was 15.3%, against 26.2% for the second-best entry. It also showed that removing any one convolutional layer made results worse.

Read it for

Section 3.1 on ReLUs and Section 4 on fighting overfitting are the ideas that lasted. Section 3.2's two-GPU split is an engineering workaround you can skim; Figure 3, the learned first-layer filters, is worth a look.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.