the open
    research
    Contribute
    Back to the catalogue

    Language models · 2017

    Attention Is All You Need

    Vaswani et al.

    Read the original
    One query, every key.

    The idea, in brief

    Replace recurrence with attention to let each token draw context from the whole sequence.

    The architecture that changed how we build language models. A closer look at attention, parallelism, and what makes a Transformer work.

    Read with a question in mind.

    What evidence supports the main claim? Which assumptions would change the result? What would it take to reproduce it?

    Go to the source

    This introduction is an editorial summary. The original paper contains the methods, experiments, and limitations you need for a closer look.

    arXiv: 1706.03762

    On the reading lists

    Why it matters

    Introduced the Transformer, the architecture underlying modern foundation models.

    Condensed from the expert lists' notes.

    Inside the paper

    The problem
    Sequence models for translation commonly used recurrent or convolutional operations to connect words across a sentence. Recurrence also limited how much of a sequence could be processed simultaneously during training.
    The idea
    Build the encoder and decoder around attention and position-wise layers, without recurrent or convolutional sequence processing. Multiple attention heads can examine different relationships in parallel.
    How it works
    • The encoder combines self-attention with a feed-forward network at each position, repeating this structure in a stack.
    • The decoder attends to earlier output positions and to encoder representations, with a mask preventing access to future output tokens.
    • Each attention head compares queries with keys and uses the resulting weights to combine values; several heads are joined together.
    • Position information is added to token embeddings, giving the model a way to distinguish ordering without recurrence.
    What it showed
    On English-to-German translation, the big model exceeds the compared systems in BLEU; on English-to-French it surpasses the compared single models with less training computation. The authors also report a constituency parsing experiment.
    Read it for
    Start with Figure 1 and Sections 3.1–3.2, then read Section 3.5 on position and Table 2 for translation results. The optimizer and batching details in Section 5 can wait.

    Written in our own words from the paper itself. A draft the editors have not yet reviewed.

    Reading it

    Difficulty: Intermediate

    Read this first: attention (Bahdanau et al.); sequence-to-sequence models.

    The record

    First released
    2017 (from arXiv 1706.03762)
    Citations
    197,211 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Computer vision & multimodal, AI for science
    Score
    0.800 · computed rank 13 · consensus 1, must-read 1, citations percentile 1 (2010s · language), awards 0, existing lists 0.5. Method

    Editor's notes

    • Readability check: pass: The Transformer; compact and readable once attention is familiar. (AI draft (Claude), not yet reviewed, 2026-10-29)

    This paper's reading-list entry has its rank, release date and sources in full.

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.