A good question opens things up.
The discussion space is being built. These prompts are a starting point for reading this paper together.
Which part of the approach is doing the most work?
What do the data and evaluation leave out?
How would you reproduce this with less compute?
These are editorial prompts, not posts from community members.
Loading discussions…
On the reading lists
- #70 in the Top 500 (2026 edition)
- #55 of 100 papers to read in your lifetime (2026 edition)
- #11 in Language, speech & LLMs (2026 edition)
Why it matters
Quantified predictable power-law relationships between model size, data, compute, and performance.
Condensed from the expert lists' notes.
Inside the paper
- The problem
- Training larger language models requires choices about model size, data and computation before the final loss is known. The authors study whether experiments at smaller scales can inform those choices.
- The idea
- Measure held-out language-model loss across systematic changes in resources, then fit simple relationships to the observed trends. The analysis also considers how to allocate a fixed training budget.
- How it works
- The experiments train Transformer language models of different sizes and evaluate prediction loss on held-out text.
- They vary the amount of training data and compute, while investigating how architectural shape affects loss at a given parameter count.
- Fitted curves relate loss to model size, data and computation, and expose the extra loss associated with overfitting.
- The authors compare model sizes along training curves to estimate which allocation of a compute budget yields the lowest loss.
- What it showed
- Across the studied range, loss follows approximate power-law trends and depends more strongly on scale than on many shape choices. The compute analysis suggests training larger models for fewer steps than needed to reach their minimum loss.
- Read it for
- Start with Figures 1–2 and Section 1.2 for the claims, then read Sections 3 and 6 to see how they are measured. The derivations in the appendices can wait.
Written in our own words from the paper itself. A draft the editors have not yet reviewed.
Reading it
Difficulty: Introductory
Read this first: the Transformer (Attention Is All You Need); log-log plots.
The record
- First released
- 2020 (from arXiv 2001.08361)
- Citations
- 9,325 citations, as counted by Papers With Code (CC BY-SA 4.0)
- Named by
- both expert lists
- Also relevant to
- Deep learning
- Score
- 0.702 · computed rank 70 · consensus 0.94, must-read 1, citations percentile 0.771 (2020s · language), awards 0, existing lists 0.5. Method
Editor's notes
- Readability check: pass: Short findings section with lasting influence on how models are built. (AI draft (Claude), not yet reviewed, 2026-10-29)
This paper's reading-list entry has its rank, release date and sources in full.
Data sources
Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.