Deep contextualized word representations

Inside the paper

The problem

A single vector for a word cannot reflect how its meaning changes between sentences. Existing task models also had to learn much of their context-sensitive representation from the labeled data available for each task.

The idea

Learn representations from a bidirectional language model and combine its internal layers for each downstream task. The resulting representation depends on the sentence around the word.

How it works

• A forward language model predicts each token from earlier text, and a backward model predicts it from later text.
• The pretrained network produces representations at multiple layers for every token in a sentence.
• A downstream model learns a weighted combination of these layers and a scaling factor, then incorporates the result as an additional representation.
• The authors add this representation to existing supervised architectures and compare performance across several language understanding tasks.

What it showed

The paper reports improvements across its evaluated tasks, including question answering and sequence labeling. Its analysis finds that different layers and insertion points contribute differently depending on the task.

Read it for

Read Sections 3.1–3.3 for the representation and how it enters a task model, then Section 5.3 for the layer analysis. The individual task architectures in Section 4 can be read selectively.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.