Attention Is All You Need

Inside the paper

The problem

Sequence models for translation commonly used recurrent or convolutional operations to connect words across a sentence. Recurrence also limited how much of a sequence could be processed simultaneously during training.

The idea

Build the encoder and decoder around attention and position-wise layers, without recurrent or convolutional sequence processing. Multiple attention heads can examine different relationships in parallel.

How it works

• The encoder combines self-attention with a feed-forward network at each position, repeating this structure in a stack.
• The decoder attends to earlier output positions and to encoder representations, with a mask preventing access to future output tokens.
• Each attention head compares queries with keys and uses the resulting weights to combine values; several heads are joined together.
• Position information is added to token embeddings, giving the model a way to distinguish ordering without recurrence.

What it showed

On English-to-German translation, the big model exceeds the compared systems in BLEU; on English-to-French it surpasses the compared single models with less training computation. The authors also report a constituency parsing experiment.

Read it for

Start with Figure 1 and Sections 3.1–3.2, then read Section 3.5 on position and Table 2 for translation results. The optimizer and batching details in Section 5 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.