wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Inside the paper

The problem

Speech recognition usually needs transcribed audio, which is expensive to collect for every setting. Unlabeled recordings are more plentiful, but the model must learn useful representations from them before seeing transcripts.

The idea

Pretrain a speech encoder by hiding parts of its audio representation and asking it to identify the corresponding learned discrete targets. Then fine-tune the network on labeled speech for recognition.

How it works

• A convolutional feature encoder turns raw audio into a sequence of latent representations, some of which are masked.
• A Transformer processes the masked sequence and produces context-sensitive representations for the hidden positions.
• A quantization module supplies discrete targets, and a contrastive objective distinguishes the correct target from distractors.
• During fine-tuning, a prediction layer is added and the model learns from transcripts using a connectionist temporal classification objective.

What it showed

On LibriSpeech, the authors report strong recognition with limited labeled data after pretraining on unlabeled audio. Their ablations examine masking and quantization, while a separate experiment evaluates phoneme recognition.

Read it for

Read Figure 1 and Sections 2–3 for the pretraining task, then Section 5.1 for low-resource results and Section 5.4 for ablations. Detailed optimizer settings in Section 4 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.