Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Inside the paper

The problem

Training a deep network requires choices of learning rate and initialization, while updates to earlier layers change the inputs seen by later ones. The authors frame this changing input distribution as a source of training difficulty.

The idea

They insert a normalization operation into the network and train through it. Each training mini-batch supplies statistics, while learned parameters restore flexibility in the resulting activations.

How it works

• For each feature, calculate the mini-batch mean and variance and use them to normalize its values.
• Apply a learned scale and offset, allowing the network to represent transformations that simple normalization alone might exclude.
• Backpropagate through the normalization calculation as well as the learned scale and offset.
• At inference, use estimates of population statistics so that an individual prediction does not depend on the other examples in its batch.
• For convolutional features, pool statistics across examples and spatial positions within each feature map.

What it showed

In the authors’ image classification experiments, a batch-normalized Inception variant reached the original model’s accuracy with far fewer training steps. Their digit experiment also tracked more stable activation distributions and faster improvement in test accuracy.

Read it for

Read Section 3 and Algorithms 1 and 2 for training and inference, then compare Figures 1 and 3 for the reported effects. The architecture-specific adjustments in Section 4.2.1 can be skimmed on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.