An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Inside the paper

The problem

Convolutional networks dominated large-scale image recognition, while applications of transformers to images often retained convolutional structure or used specialized attention patterns. The authors ask whether a standard transformer can learn directly from images.

The idea

The vision transformer turns an image into a sequence of patch representations and applies a transformer encoder to that sequence. Large-scale supervised pretraining supplies the data needed for transfer to smaller recognition tasks.

How it works

• An image is split into fixed-size patches, which are flattened and projected into vectors of a common dimension.
• Position embeddings are added so the model can use information about where each patch came from.
• A learned classification token joins the patch sequence, and the transformer processes the tokens using self-attention and feed-forward layers.
• After pretraining on image labels, the model is fine-tuned for a downstream dataset with a task-specific classification head.

What it showed

The experiments show that pretraining data size matters: the transformer trails comparable convolutional models with less data but performs strongly after larger-scale pretraining. The authors also inspect learned position embeddings and attention patterns.

Read it for

Section 3.1 and Figure 1 give the model in one place; Section 4.3 makes the data-size comparison explicit. You can skim Section 4.6 on self-supervision if your immediate interest is the supervised model.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.