Mastering the game of Go with deep neural networks and tree search

Inside the paper

The problem

Go has too many possible move sequences to search exhaustively, and evaluating a position is difficult. Earlier strong programs used search and human-inspired features but did not reach professional strength on the full board.

The idea

AlphaGo combines learned move preferences and position evaluations with a search that examines promising continuations. Its training starts with expert games and then uses self-play to improve the policy and train a value network.

How it works

• A supervised policy network learns to predict moves from expert positions, and a faster policy is trained for simulated rollouts.
• The supervised network is further trained through self-play to increase the chance of winning against previous versions of itself.
• A separate value network learns to predict game outcomes from positions sampled from self-play games.
• During tree search, policy probabilities guide which moves receive attention, while network evaluations and fast rollouts help estimate the value of positions.

What it showed

The system defeated the European champion Fan Hui by five games to none and won 99.8% of games against other Go programs in the reported evaluation. The paper also found that its reinforcement-trained policy beat its supervised starting point in head-to-head play.

Read it for

Figure 1 gives the training sequence and Figure 3 shows the search; read those alongside the sections on policy and value networks. Skim the detailed distributed-search implementation in Methods until you need to reproduce the system.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.