the open
    research
    Contribute
    Back to the catalogue

    Language models · 2017

    Attention Is All You Need

    Vaswani et al.

    Read the original
    One query, every key.

    A good question opens things up.

    The discussion space is being built. These prompts are a starting point for reading this paper together.

    Reading prompt 1

    Which part of the approach is doing the most work?

    Reading prompt 2

    What do the data and evaluation leave out?

    Reading prompt 3

    How would you reproduce this with less compute?

    These are editorial prompts, not posts from community members.

    Loading discussions…

    On the reading lists

    Why it matters

    Introduced the Transformer, the architecture underlying modern foundation models.

    Condensed from the expert lists' notes.

    Inside the paper

    The problem
    Sequence models for translation commonly used recurrent or convolutional operations to connect words across a sentence. Recurrence also limited how much of a sequence could be processed simultaneously during training.
    The idea
    Build the encoder and decoder around attention and position-wise layers, without recurrent or convolutional sequence processing. Multiple attention heads can examine different relationships in parallel.
    How it works
    • The encoder combines self-attention with a feed-forward network at each position, repeating this structure in a stack.
    • The decoder attends to earlier output positions and to encoder representations, with a mask preventing access to future output tokens.
    • Each attention head compares queries with keys and uses the resulting weights to combine values; several heads are joined together.
    • Position information is added to token embeddings, giving the model a way to distinguish ordering without recurrence.
    What it showed
    On English-to-German translation, the big model exceeds the compared systems in BLEU; on English-to-French it surpasses the compared single models with less training computation. The authors also report a constituency parsing experiment.
    Read it for
    Start with Figure 1 and Sections 3.1–3.2, then read Section 3.5 on position and Table 2 for translation results. The optimizer and batching details in Section 5 can wait.

    Written in our own words from the paper itself. A draft the editors have not yet reviewed.

    Reading it

    Difficulty: Intermediate

    Read this first: attention (Bahdanau et al.); sequence-to-sequence models.

    The record

    First released
    2017 (from arXiv 1706.03762)
    Citations
    197,211 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Computer vision & multimodal, AI for science
    Score
    0.800 · computed rank 13 · consensus 1, must-read 1, citations percentile 1 (2010s · language), awards 0, existing lists 0.5. Method

    Editor's notes

    • Readability check: pass: The Transformer; compact and readable once attention is familiar. (AI draft (Claude), not yet reviewed, 2026-10-29)

    This paper's reading-list entry has its rank, release date and sources in full.

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.