the open
    research
    Contribute
    Back to the catalogue

    Computer vision · 2020

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy et al.

    Read the original
    Cut the image into words.

    The idea, in brief

    Feed image patches into a Transformer to learn visual representations at scale.

    Treat an image as a sequence of patches, and revisit the role of convolution in visual recognition.

    Read with a question in mind.

    What evidence supports the main claim? Which assumptions would change the result? What would it take to reproduce it?

    Go to the source

    This introduction is an editorial summary. The original paper contains the methods, experiments, and limitations you need for a closer look.

    arXiv: 2010.11929

    On the reading lists

    Why it matters

    Demonstrated that pure transformers can rival CNNs, establishing the vision transformer paradigm.

    Condensed from the expert lists' notes.

    Inside the paper

    The problem
    Convolutional networks dominated large-scale image recognition, while applications of transformers to images often retained convolutional structure or used specialized attention patterns. The authors ask whether a standard transformer can learn directly from images.
    The idea
    The vision transformer turns an image into a sequence of patch representations and applies a transformer encoder to that sequence. Large-scale supervised pretraining supplies the data needed for transfer to smaller recognition tasks.
    How it works
    • An image is split into fixed-size patches, which are flattened and projected into vectors of a common dimension.
    • Position embeddings are added so the model can use information about where each patch came from.
    • A learned classification token joins the patch sequence, and the transformer processes the tokens using self-attention and feed-forward layers.
    • After pretraining on image labels, the model is fine-tuned for a downstream dataset with a task-specific classification head.
    What it showed
    The experiments show that pretraining data size matters: the transformer trails comparable convolutional models with less data but performs strongly after larger-scale pretraining. The authors also inspect learned position embeddings and attention patterns.
    Read it for
    Section 3.1 and Figure 1 give the model in one place; Section 4.3 makes the data-size comparison explicit. You can skim Section 4.6 on self-supervision if your immediate interest is the supervised model.

    Written in our own words from the paper itself. A draft the editors have not yet reviewed.

    Reading it

    Difficulty: Intermediate

    Read this first: the Transformer (Attention Is All You Need); convolutional networks (LeNet).

    The record

    First released
    2020 (from arXiv 2010.11929)
    Published
    2021
    Citations
    71,638 citations, as counted by Papers With Code (CC BY-SA 4.0)
    Named by
    both expert lists
    Score
    0.723 · computed rank 48 · consensus 0.91, must-read 1, citations percentile 1 (2020s · vision), awards 0, existing lists 0. Method

    Editor's notes

    • Readability check: pass: ViT: shows the Transformer applies to images, simply. (AI draft (Claude), not yet reviewed, 2026-10-29)

    This paper's reading-list entry has its rank, release date and sources in full.

    Data sources

    Catalogue data from OpenAlex (CC0), Crossref (CC0 metadata; no abstracts are shown), arXiv, and Papers With Code (CC BY-SA 4.0). Each count names its source. See the method.