Sequence to Sequence Learning with Neural Networks

Inside the paper

The problem

A neural network normally expects inputs and outputs of fixed dimensions, while a translated sentence can be longer or shorter than its source. Learning such mappings without a task-specific alignment was an open challenge.

The idea

Use one recurrent network to summarize the input and another to generate the output, training both on paired sequences. The paper uses long short-term memory networks for this mapping.

How it works

• The encoder reads the source tokens and passes its final state to a separate decoder as a representation of the sentence.
• The decoder predicts one target token at a time, conditioning on its state and the preceding target tokens.
• Training maximizes the probability of target sentences given their sources, and decoding uses beam search to explore candidate outputs.
• The authors reverse the source sentence during training and testing, shortening the distance between the beginning of each source and its translation.

What it showed

On English-to-French translation, the network achieves a strong BLEU score and improves a conventional system when used to rescore its candidates. Reversing the source substantially improves its results.

Read it for

Read Section 2 for the model and Section 3.3 for the reversal experiment, then look at Section 3.6 for the comparison. Section 3.5 on parallel training can be skimmed on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.