Long Short-Term Memory

Inside the paper

The problem

A recurrent network can carry information from earlier inputs, but training it to use that information after a long delay is difficult. In conventional recurrent training, error signals passed back over many steps can shrink or grow rapidly.

The idea

The paper combines memory cells with gates that control when information enters and leaves. A fixed self-connection gives each cell an internal path along which error can persist across time.

How it works

• A cell keeps an internal state that carries forward from one time step to the next through a fixed self-connection.
• An input gate scales new information before it is added to that state, helping protect stored information from irrelevant inputs.
• An output gate controls how much of the cell’s state is exposed to the rest of the network.
• The learning algorithm adjusts weights using error signals while truncating certain gradient paths, preserving the constant error path through the cell.

What it showed

On artificial sequence tasks, the authors report more successful runs and faster learning than the compared recurrent methods. Their adding experiment also tests whether the network can retain marked real values over long sequences and combine them at the end.

Read it for

Start with Section 3 for the difficulty of long delays, then use Figure 1 and Section 4 to follow the cell and its gates. Section 5.4 gives a concrete task; the full update equations in Appendix A can wait for a closer reading.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.