Robust Speech Recognition via Large-Scale Weak Supervision

Inside the paper

The problem

Speech recognizers often perform well on a benchmark close to their training data but degrade on unfamiliar recording conditions or speakers. The paper examines whether broad, naturally occurring audio and transcript pairs can improve robustness.

The idea

Train an encoder-decoder model on a large collection of audio paired with transcripts gathered from the web. Task and language markers allow the same model to perform several speech-related tasks.

How it works

• Audio is converted to log-Mel spectrograms and processed by an encoder, while a decoder generates text tokens autoregressively.
• Training examples include transcriptions and translations, with special tokens indicating the language and task when applicable.
• The data pipeline filters audio and text pairs to reduce mismatches, duplicates and other sources of noisy supervision.
• At inference, the paper uses decoding heuristics for long recordings and compares models without benchmark-specific fine-tuning.

What it showed

The evaluated models transfer well across several speech recognition datasets, and the paper reports multilingual recognition and translation results. Error rates still vary substantially across languages and recording conditions.

Read it for

Start with the data and architecture descriptions on pages 2–4, then examine the recognition comparisons and language breakdown in the results. The extensive per-language tables can wait until you need a particular benchmark.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.