Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Inside the paper

The problem

Large neural networks can fit their training data closely while performing less well on new examples. Averaging many separately trained networks is one response, but it can be expensive at prediction time.

The idea

Dropout trains a network with randomly chosen units temporarily removed on each presentation of data. The paper treats the resulting collection of thinned networks as an approximation to a large ensemble.

How it works

• Choose which units to retain using independent random draws, and omit the dropped units and their connections for that training update.
• Compute predictions and gradients through the remaining network, updating the weights that participated in that pass.
• Resample the retained units for subsequent training examples, so different subsets share and update the same underlying weights.
• At test time, use the full network with appropriately scaled weights to approximate averaging predictions across the thinned networks.

What it showed

The authors report improved generalization over networks without dropout on the datasets they study, including image, speech and text tasks. They also compare the test-time scaling approximation with sampling multiple dropout networks.

Read it for

Read Sections 2 and 3 for the mechanism and ensemble interpretation, then Section 6 for the experiments. Section 7.5 is useful for understanding the approximation; the later restricted Boltzmann machine discussion can be skimmed initially.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.