Neural Machine Translation by Jointly Learning to Align and Translate

Inside the paper

The problem

Translation networks compressed an entire source sentence into one vector before producing the target. That representation made longer inputs especially difficult to carry through the decoder.

The idea

The authors let the decoder consult a sequence of source representations anew for each target word. A learned alignment assigns weight to the relevant positions while the translation model is trained.

How it works

• A recurrent encoder reads the source in both directions, combining its states to represent each source position in context.
• At each output step, a small network scores each source position against the current state of the decoder.
• The scores become weights for a weighted sum of the source representations, providing a context for the next word.
• The alignment and translation components learn together from parallel sentences; beam search approximates the best translation at test time.

What it showed

On English-to-French news translation, the proposed models outperform the authors’ fixed-vector comparison and handle longer sentences more robustly. Their visualized alignments also connect many target words with plausible source positions.

Read it for

Start with Figure 1 and Sections 3.1–3.2 to trace the information flow, then compare sentence lengths in Figure 2 and the examples in Figure 3. The implementation details in the appendices can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.