Scaling Laws for Neural Language Models

Inside the paper

The problem

Training larger language models requires choices about model size, data and computation before the final loss is known. The authors study whether experiments at smaller scales can inform those choices.

The idea

Measure held-out language-model loss across systematic changes in resources, then fit simple relationships to the observed trends. The analysis also considers how to allocate a fixed training budget.

How it works

• The experiments train Transformer language models of different sizes and evaluate prediction loss on held-out text.
• They vary the amount of training data and compute, while investigating how architectural shape affects loss at a given parameter count.
• Fitted curves relate loss to model size, data and computation, and expose the extra loss associated with overfitting.
• The authors compare model sizes along training curves to estimate which allocation of a compute budget yields the lowest loss.

What it showed

Across the studied range, loss follows approximate power-law trends and depends more strongly on scale than on many shape choices. The compute analysis suggests training larger models for fewer steps than needed to reach their minimum loss.

Read it for

Start with Figures 1–2 and Section 1.2 for the claims, then read Sections 3 and 6 to see how they are measured. The derivations in the appendices can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.