the open
    research
    Contribute
    Back to the catalogue

    Training & compute · 2020

    Scaling Laws for Neural Language Models

    Kaplan et al.

    Read the original
    Straight lines on log–log axes.

    The idea, in brief

    Study performance as a function of model size, dataset size, and training compute.

    How model size, data, and compute shape performance—and why the assumptions behind a scaling curve matter.

    Read with a question in mind.

    What evidence supports the main claim? Which assumptions would change the result? What would it take to reproduce it?

    Go to the source

    This introduction is an editorial summary. The original paper contains the methods, experiments, and limitations you need for a closer look.

    arXiv: 2001.08361

    On the reading lists

    Why it matters

    Quantified predictable power-law relationships between model size, data, compute, and performance.

    Condensed from the expert lists' notes.

    Inside the paper

    The problem
    Training larger language models requires choices about model size, data and computation before the final loss is known. The authors study whether experiments at smaller scales can inform those choices.
    The idea
    Measure held-out language-model loss across systematic changes in resources, then fit simple relationships to the observed trends. The analysis also considers how to allocate a fixed training budget.
    How it works
    • The experiments train Transformer language models of different sizes and evaluate prediction loss on held-out text.
    • They vary the amount of training data and compute, while investigating how architectural shape affects loss at a given parameter count.
    • Fitted curves relate loss to model size, data and computation, and expose the extra loss associated with overfitting.
    • The authors compare model sizes along training curves to estimate which allocation of a compute budget yields the lowest loss.
    What it showed
    Across the studied range, loss follows approximate power-law trends and depends more strongly on scale than on many shape choices. The compute analysis suggests training larger models for fewer steps than needed to reach their minimum loss.
    Read it for
    Start with Figures 1–2 and Section 1.2 for the claims, then read Sections 3 and 6 to see how they are measured. The derivations in the appendices can wait.

    Written in our own words from the paper itself. A draft the editors have not yet reviewed.

    Reading it

    Difficulty: Introductory

    Read this first: the Transformer (Attention Is All You Need); log-log plots.

    The record

    First released
    2020 (from arXiv 2001.08361)
    Citations
    9,325 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Also relevant to
    Deep learning
    Score
    0.702 · computed rank 70 · consensus 0.94, must-read 1, citations percentile 0.771 (2020s · language), awards 0, existing lists 0.5. Method

    Editor's notes

    • Readability check: pass: Short findings section with lasting influence on how models are built. (AI draft (Claude), not yet reviewed, 2026-10-29)

    This paper's reading-list entry has its rank, release date and sources in full.

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.