WaveNet: A Generative Model for Raw Audio

Inside the paper

The problem

Speech synthesis systems often relied on a separate signal-processing stage to convert predicted acoustic features into sound. The authors investigate whether a network can instead model audio samples directly.

The idea

WaveNet assigns a probability to each audio sample given earlier samples, using causal convolutions to represent the preceding waveform. Conditioning can guide the generated sound toward a speaker or linguistic input.

How it works

• The waveform is quantized and the network predicts a distribution for the next sample rather than a single fixed value.
• Causal convolutions prevent a prediction from seeing future samples, while dilated layers expand the range of past audio available to it.
• Gated activations and residual and skip connections help combine information through a deep stack of convolutional layers.
• For speech synthesis, conditioning features are supplied to the network; generation proceeds sample by sample from its predicted distributions.

What it showed

Listening tests report better naturalness for WaveNet speech than the compared synthesis systems in English and Mandarin. The paper also presents music generation and an experiment applying related convolutional layers to speech recognition.

Read it for

Read Figure 2 and Sections 2.1–2.5 to follow the architecture, then Table 1 and Section 3.2 for the speech results. The recognition experiment in Section 3.4 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.