Understanding the difficulty of training deep feedforward neural networks

Inside the paper

The problem

Deep feedforward networks trained from ordinary random initializations often learned poorly, even when shallower networks were easier to optimize. The authors investigate what happens to activations and gradients as they pass through layers.

The idea

The paper connects training difficulties to saturating nonlinearities and to changes in signal scale across layers. It proposes an initialization that accounts for both the number of inputs and outputs of each layer.

How it works

• Train networks of different depths and inspect activation distributions as learning proceeds, rather than looking only at final error.
• Compare sigmoid, hyperbolic tangent and softsign nonlinearities to see how their activations enter or leave saturated regions.
• Analyze the propagation of activation and gradient variance at initialization under simplifying assumptions.
• Scale randomly initialized weights using the sizes of the adjacent layers, then compare the resulting gradients and learning curves with the standard initialization.

What it showed

The experiments show long plateaus with sigmoid networks and differences in how the alternative nonlinearities saturate. The proposed initialization improves gradient propagation and training in the paper’s comparisons, though its derivation relies on approximations.

Read it for

Start with Section 3 and its activation plots, then read Section 4.2 and Figure 6 for initialization. Section 4.3 follows gradients during learning; the derivation in Section 4.2.1 can be revisited more slowly.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.