Learning Transferable Visual Models From Natural Language Supervision

Inside the paper

The problem

Image classifiers typically learned a fixed set of labeled categories, making new categories costly to add. Earlier work using text associated with images had shown promise but lagged on common benchmarks.

The idea

CLIP learns to match images with their associated text, using a large collection of web image-text pairs. At evaluation time, text descriptions can specify the categories to compare with an image.

How it works

• Separate image and text encoders map the two inputs into a shared space where their similarity can be measured.
• During training, matching pairs in a batch are encouraged to score above mismatched combinations of images and text.
• For a new classification task, class names are placed in text prompts and encoded to form candidate class representations.
• The image encoder produces a representation of the test image, which is compared with the candidate text representations to choose a class.

What it showed

The authors evaluate zero-shot transfer across a broad collection of vision datasets, finding strong results on some tasks and substantial weaknesses on others. They also examine distribution shifts and the effects of prompt wording.

Read it for

Figure 1 and Sections 2.3 and 3.1.2 explain training and zero-shot use; Sections 3.1.4–3.1.5 show why prompts and task choice matter. The scaling details in Section 2.4 can wait on a first reading.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.