Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Inside the paper

The problem

Aligning a language model with preferences often involves fitting a reward model and then optimizing the generator against it. That pipeline introduces several stages and can be difficult to tune.

The idea

The authors derive a preference objective that trains the language model directly from pairs of preferred and disfavored responses. A reference model anchors the update through the relative probabilities of the two responses.

How it works

• Begin with a reference language model and a dataset in which a preferred response is paired with a less preferred one for the same prompt.
• Express the reward associated with a response through its probability under the trained model relative to the reference model.
• Use the resulting pairwise classification loss to increase the relative preference for chosen responses while accounting for the reference.
• Optimize this loss directly on preference pairs, then compare the resulting model with supervised and reinforcement-learning baselines.

What it showed

The experiments compare preference optimization on controlled sentiment, summarization and dialogue tasks. The paper reports competitive or stronger preference judgments in these settings and checks automated judgments against human evaluations.

Read it for

Read Sections 3–4 for the derivation and Figure 1 for the training comparison, then Sections 6.2 and 6.4 for evaluation. The longer proofs in the appendix can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.