Reducing the Dimensionality of Data with Neural Networks

Inside the paper

The problem

Principal components analysis compresses data using linear directions, but some datasets have nonlinear structure. Deep autoencoders could in principle learn nonlinear codes, yet their weights were difficult to optimize from random starting values.

The idea

The authors initialize a deep autoencoder by learning one layer at a time, then fine-tune the complete network to reconstruct its input. A narrow middle layer provides the compressed representation.

How it works

• Train a restricted Boltzmann machine to learn features from the input, then use its feature activations as data for the next layer.
• Repeat this layerwise procedure to build a stack of features, with a small code layer at the top.
• Unroll the stack into an encoder and a decoder, initializing the decoder from the learned weights in reverse order.
• Fine-tune the complete network with backpropagation to reduce the difference between the original input and its reconstruction.

What it showed

On synthetic curves, handwritten digits, and face patches, the reported autoencoders reconstruct test inputs more accurately than the paper’s PCA comparisons. The paper also evaluates compressed document representations for retrieval.

Read it for

Figure 1 and pages 505–506 explain pretraining and the unrolled network; Figures 2–4 show what the codes preserve. The energy equations on pages 505–506 can be skimmed initially and revisited for implementation.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.