Proximal Policy Optimization Algorithms

Inside the paper

The problem

Policy-gradient methods can take updates that change the policy too much, while trust-region methods add implementation and computational complexity. The authors wanted a practical objective that supports several passes over collected experience.

The idea

PPO modifies a policy-gradient surrogate so that large changes in action probabilities lose their incentive to improve the objective. The paper also considers an adaptive penalty based on the difference between old and new policies.

How it works

• Collect trajectory segments using the current policy and estimate how much better each sampled action was than the expected value of its state.
• Form a ratio between the new and old probabilities of each sampled action, and multiply it by the estimated advantage.
• For the clipped objective, compare this term with a version whose ratio is clipped, and optimize the smaller value.
• Run multiple minibatch updates over the collected data, then use the updated policy to gather a fresh batch.

What it showed

On the paper’s simulated continuous-control benchmark, the clipped objective performed better in the reported aggregate comparison than the tested penalty alternatives. In Atari experiments, PPO led on more games under the training-period score, while ACER led under the final-performance score.

Read it for

Read Section 3 and Figure 1 to see what clipping changes, then Algorithm 1 and Tables 1–2 for the procedure and comparisons. Section 4’s alternative penalty can be skimmed initially.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.