Neural Machine Translation of Rare Words with Subword Units

Inside the paper

The problem

Word-based neural translation systems use a limited vocabulary, so infrequent words and names may become unknown tokens. A dictionary fallback can help, but it does not reliably form new words or transliterate names.

The idea

Represent words as sequences of reusable pieces and let the translation model generate those pieces. The authors adapt byte pair encoding to learn a compact vocabulary of variable-length units.

How it works

• Start with characters and repeatedly join the most frequent adjacent pair, recording each merge as a reusable segmentation rule.
• Apply the learned merges to training and test text, preserving word boundaries so output pieces can be joined back into words.
• Train an attention-based translation model on the resulting sequences instead of treating every complete word as a separate symbol.
• The paper compares independently learned segmentation with a joint vocabulary across languages, as well as word and character-based alternatives.

What it showed

In English-to-German and English-to-Russian experiments, subword systems improve rare-word measures over the dictionary-backed baseline. Examples show both successful transliteration and errors caused by inconsistent segmentation.

Read it for

Read Section 3.2 for the merge procedure and Tables 2–3 for the translation comparisons. Section 5.2 gives concrete examples; detailed training settings in Section 4 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.