Language Models are Few-Shot Learners

Inside the paper

The problem

Adapting pretrained language models to a new task often required a separate labeled dataset and a fine-tuning run. The authors ask how far a model can go when a task is specified in text at inference time.

The idea

Train a family of autoregressive language models and evaluate them using instructions and examples placed in the input. The model continues the prompt without changing its parameters for the task.

How it works

• The models learn to predict the next token from text drawn from a mixture of sources, including filtered web data.
• At evaluation time a zero-shot prompt describes the task, while one-shot and few-shot prompts also supply examples.
• The model completes a new example from the same prompt; the authors compare several model sizes and prompt conditions.
• They check benchmark overlap with training data and report limitations where contamination could affect the interpretation of results.

What it showed

The largest model performs well on several question answering, translation and completion tests, although it remains weak on some tasks. Few-shot gains often grow with model size, but performance and contamination vary by benchmark.

Read it for

Figure 2.1 makes the evaluation settings concrete; read Sections 3.2–3.3 for examples and Section 4 for contamination. The full list of prompts in the appendix is useful later.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.