Training language models to follow instructions with human feedback

Inside the paper

The problem

Predicting the next piece of text does not by itself ensure that a language model follows a user’s instructions, tells the truth or avoids harmful responses.

The idea

The authors adapt a pretrained language model using examples and comparisons provided by human labelers, then evaluate its responses on held-out prompts and public tasks.

How it works

• Collect demonstrations of desired responses to prompts and fine-tune the pretrained model to imitate those demonstrations.
• Ask labelers to rank several model responses to a prompt, then train a reward model to predict their preferences.
• Use reinforcement learning to improve the supervised model against the learned reward, while limiting how far it moves from the starting policy.
• In one variant, mix in a pretraining objective during reinforcement learning to reduce losses on some public language tasks.

What it showed

On the paper’s API prompt distribution, labelers preferred outputs from the smaller InstructGPT model to those from a much larger GPT-3 baseline. The models still made mistakes, and the authors found no significant improvement on the bias datasets they examined.

Read it for

Begin with Figure 2 and Sections 3.1 and 3.4 for the training and labelers, then read Sections 4.1–4.3 for results and examples. The detailed dataset and hyperparameter appendices can be skimmed initially.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.