Human-level control through deep reinforcement learning

Inside the paper

The problem

Learning to act from game images requires extracting useful features as well as choosing actions. Combining neural networks with reinforcement learning can be unstable because consecutive observations and changing targets are correlated.

The idea

DQN learns action values with a convolutional network, using stored experience and a periodically refreshed target network to make training more stable.

How it works

• The network receives a short stack of preprocessed game frames and produces a value estimate for each available action.
• The agent usually chooses the action with the highest estimated value but sometimes explores by choosing randomly.
• It stores transitions containing the state, action, reward and next state, then trains on randomly sampled batches from that memory.
• Each update moves the selected action value toward the observed reward plus the discounted best value at the next state, calculated by a separate, periodically updated network.

What it showed

With the same architecture and settings across the evaluation, DQN surpassed the best earlier reinforcement-learning methods on 43 of 49 Atari games. It scored more than 75% of the human tester’s level on 29 games, leaving substantial differences between games.

Read it for

Start with Figure 1 and the first page for the learning procedure, then inspect Figure 3 for the uneven game-by-game results. The Methods section gives the implementation details and can be skimmed on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.