Deep reinforcement learning from human preferences

Inside the paper

The problem

Designing a numerical reward for a complex behavior can be difficult, even when a person can recognize which of two attempts is better.

The idea

The paper trains an agent using a learned reward model built from human comparisons of short clips, while continuing to improve the policy and request feedback.

How it works

• Run the current policy to collect short segments of behavior, then present pairs of segments for a person to compare.
• Train a reward predictor so that the total predicted reward of the preferred segment tends to exceed that of the other segment.
• Use the learned reward in a reinforcement learning algorithm to update the policy, generating new behavior for further comparisons.
• Select comparisons using disagreement among reward predictors, and keep updating the reward model as the agent’s behavior changes.

What it showed

The experiments learned a range of simulated robotics and Atari behaviors with comparison feedback, including a backflip specified by human judgments. Performance varied across tasks, and ablations examined the contribution of components of the method.

Read it for

Start with Figure 1 and Section 2.2 for the feedback loop, then Section 3.2 and Figure 4 for the backflip. The detailed ablations in Section 3.3 can be skimmed before returning to the experimental comparisons.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.