Policy Gradient Methods for Reinforcement Learning with Function Approximation

Inside the paper

The problem

With function approximation, choosing actions greedily from estimated values can make the policy change abruptly and complicate convergence. Researchers needed a way to improve a directly parameterized policy while still using an approximate value function.

The idea

The paper derives an expression for the policy gradient and identifies conditions under which an approximate action-value function can replace the true one without changing that gradient.

How it works

• Represent the policy as a differentiable function that assigns probabilities to actions in each state.
• Express the performance gradient using state visitation, policy derivatives and action values, so it can be estimated from experience.
• Fit a value approximation whose derivatives are compatible with the policy and whose weighted approximation error satisfies the paper’s condition.
• Use the resulting gradient to improve the policy; state-dependent baselines can change estimator variance without changing the expected gradient.

What it showed

The authors prove the gradient identity for average-reward and start-state formulations and give a convergence result for a form of policy iteration under stated assumptions. The result concerns an idealized update and should not be read as a guarantee for every practical actor-critic implementation.

Read it for

Read Section 1 for the theorem, Section 2 for compatibility and Section 3 for the connection to advantages. The appendix proof can be skimmed before returning to Section 4’s convergence conditions.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.