the open
    research
    Contribute
    Back to the catalogue

    Language models · 2018

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin et al.

    Read the original
    Fill the gap from both sides.

    The idea, in brief

    Pre-train bidirectional representations with masked language modeling, then fine-tune for a task.

    Learning representations from both directions of a sentence, then adapting them to language understanding tasks.

    Read with a question in mind.

    What evidence supports the main claim? Which assumptions would change the result? What would it take to reproduce it?

    Go to the source

    This introduction is an editorial summary. The original paper contains the methods, experiments, and limitations you need for a closer look.

    arXiv: 1810.04805

    On the reading lists

    Why it matters

    Established bidirectional Transformer pretraining and fine-tuning as the dominant NLP paradigm.

    Condensed from the expert lists' notes.

    Inside the paper

    The problem
    Pretrained language representations had improved language tasks, but some approaches restricted each token to one direction of context. The authors wanted a general representation that could use both sides of a token during pretraining.
    The idea
    Train a bidirectional Transformer on corrupted text and sentence relationships, then adapt its parameters to a supervised task. The same pretrained model can be used with a small task-specific output layer.
    How it works
    • Text is tokenized and packed into one or two segments, with special markers identifying the start and segment boundaries.
    • During pretraining, selected tokens are hidden or otherwise altered, and the model predicts their original identities from surrounding context.
    • A second pretraining task asks whether one segment follows another in the original text.
    • For a downstream task, an appropriate output layer is added and the whole network is fine-tuned on labeled examples.
    What it showed
    The paper reports improvements across sentence classification and question answering benchmarks, and its ablations show contributions from the pretraining tasks. The size comparison also finds gains from the larger model on the evaluated tasks.
    Read it for
    Begin with Figure 1 and Section 3, then compare the pretraining ablations in Section 5.1. The individual benchmark descriptions in Section 4 can be read selectively.

    Written in our own words from the paper itself. A draft the editors have not yet reviewed.

    Reading it

    Difficulty: Intermediate

    Read this first: the Transformer (Attention Is All You Need).

    The record

    First released
    2018 (from arXiv 1810.04805)
    Citations
    122,869 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Computer vision & multimodal, Search, retrieval & recommendation, AI for science
    Score
    0.743 · computed rank 37 · consensus 0.99, must-read 1, citations percentile 0.989 (2010s · language), awards 0, existing lists 0. Method

    Editor's notes

    • Readability check: pass: Pre-training and fine-tuning, explained clearly. (AI draft (Claude), not yet reviewed, 2026-10-29)

    This paper's reading-list entry has its rank, release date and sources in full.

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.