BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Inside the paper

The problem

Pretrained language representations had improved language tasks, but some approaches restricted each token to one direction of context. The authors wanted a general representation that could use both sides of a token during pretraining.

The idea

Train a bidirectional Transformer on corrupted text and sentence relationships, then adapt its parameters to a supervised task. The same pretrained model can be used with a small task-specific output layer.

How it works

• Text is tokenized and packed into one or two segments, with special markers identifying the start and segment boundaries.
• During pretraining, selected tokens are hidden or otherwise altered, and the model predicts their original identities from surrounding context.
• A second pretraining task asks whether one segment follows another in the original text.
• For a downstream task, an appropriate output layer is added and the whole network is fine-tuned on labeled examples.

What it showed

The paper reports improvements across sentence classification and question answering benchmarks, and its ablations show contributions from the pretraining tasks. The size comparison also finds gains from the larger model on the evaluated tasks.

Read it for

Begin with Figure 1 and Section 3, then compare the pretraining ablations in Section 5.1. The individual benchmark descriptions in Section 4 can be read selectively.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.