Distilling the Knowledge in a Neural Network

Inside the paper

The problem

An ensemble can improve predictions, but evaluating many large models for every request is expensive. A smaller model needs a way to learn from the ensemble’s behavior rather than only from hard class labels.

The idea

The authors train a compact model to match softened predictions from a larger model or ensemble. The relative probabilities of alternative classes provide information that a single correct label does not.

How it works

• Train a larger model or ensemble first, then evaluate it on a transfer set to produce target probability distributions.
• Raise the softmax temperature to reveal more of the relative preference among classes beyond the most likely answer.
• Train the smaller model against those soft targets at the same temperature, optionally combining that objective with ordinary labelled training.
• Return the smaller model to the usual softmax temperature when it is deployed for prediction.

What it showed

The paper reports digit recognition experiments and a speech recognition experiment in which a single distilled acoustic model improved on the baseline. It also explores an ensemble of specialists assigned groups of classes that a general model tends to confuse.

Read it for

Section 2 explains the transfer objective and temperature; read Sections 3 and 4 with their result tables for concrete examples. Section 5’s specialist ensemble is a separate extension that can wait on a first reading.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.