Constitutional AI: Harmlessness from AI Feedback

Inside the paper

The problem

Training assistants to avoid harmful responses had relied on human preference labels, and the resulting systems could become evasive even when a useful explanation was possible.

The idea

The authors use written principles to guide model-generated critiques, revisions and preference judgments, supplying supervision for harmlessness while retaining human feedback for helpfulness.

How it works

• A helpful assistant first answers prompts designed to elicit harmful behavior, then critiques and revises each answer using a selected principle.
• A language model is fine-tuned on the revised answers, giving it a starting policy for the next stage of training.
• The assistant produces pairs of responses, and another model selects the preferable response according to a constitutional principle.
• A preference model learns from those AI judgments and human helpfulness comparisons, and reinforcement learning updates the assistant against its scores.

What it showed

In the reported crowdworker comparisons, the resulting assistants improved the harmlessness and helpfulness tradeoff relative to the human-feedback baselines studied. The paper also found that critiques improved revisions compared with revising directly.

Read it for

Start with Section 1.2 and Figure 1 for the two stages, then Sections 3.5 and 4.3 for the comparisons. The scaling experiments in Sections 2 and 3.4 can be read later.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.