Mastering the game of Go without human knowledge

Inside the paper

The problem

The earlier AlphaGo system relied on expert games and specialized features as well as self-play. The authors asked whether a strong Go player could learn from the rules and its own experience instead.

The idea

AlphaGo Zero repeatedly improves a single network through games against itself. Search uses the network to choose moves, and the resulting searches and game outcomes become training targets for the next network.

How it works

• The network takes a representation of the board and predicts both move probabilities and the eventual winner from the current position.
• At each move, tree search uses the network’s move probabilities and position evaluations to guide simulations and form a stronger move distribution.
• Self-play games supply board positions, search distributions and final outcomes; the network is trained to match the latter two.
• The updated network guides subsequent searches, creating a loop between search, self-play and learning without rollout simulations.

What it showed

After the reported three-day training run, AlphaGo Zero defeated the version of AlphaGo that played Lee Sedol by 100 games to 0 under the stated match conditions. A comparison with a human-game-trained network found that predicting expert moves more accurately did not translate to stronger play in that test.

Read it for

Read Figures 1–2 for the learning and search loop, then Section 2 and Figure 3 for the comparisons. The detailed Methods can be skimmed until you need the training and evaluation settings.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.