Learning representations by back-propagating errors

Inside the paper

The problem

A network with hidden units can express relationships that direct input-to-output connections cannot. The challenge is deciding how to change hidden connections when the desired answers specify only the output.

The idea

The authors propagate derivatives of output error backwards through the network, giving each connection a direction in which to adjust. This lets hidden units develop internal representations suited to a task.

How it works

• Present an input and compute each unit’s output from its weighted inputs and nonlinear activation function.
• Compare output activations with their desired values to calculate an error for the example.
• Apply the chain rule backwards from the output to hidden units, calculating how each unit and connection contributes to error.
• Accumulate weight derivatives over the examples and adjust weights downhill, using a momentum term to carry some of the preceding update forward.

What it showed

The paper demonstrates networks learning mirror symmetry and family relationships, and examines the structure of the representations they form. Figure 1 shows a symmetry solution in which paired input weights have opposite signs.

Read it for

Start on the first page of the article with the forward and backward equations, then examine Figures 1 and 2 and the accompanying examples. The detailed weight-update algebra can be revisited after you see what the hidden units learn.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.