Masked Autoencoders Are Scalable Vision Learners

Inside the paper

The problem

Self-supervised pretraining had succeeded in language, but reconstructing missing image content faced a different challenge: nearby pixels can make small gaps too easy to fill. Large vision models also demand substantial training resources.

The idea

A masked autoencoder hides much of an image and learns to reconstruct the missing patches. Its encoder processes only visible patches, while a smaller decoder handles reconstruction.

How it works

• The image is split into patches and a random subset is withheld before the encoder sees the input.
• The encoder processes the remaining patches; mask tokens are added later so the decoder has positions for the missing content.
• The decoder predicts pixels for the patches, and the reconstruction loss is calculated on the masked patches.
• After pretraining, the decoder is discarded and the encoder can be adapted to recognition tasks using complete images.

What it showed

The experiments find that a high masking ratio works well for this setup and that the asymmetric design reduces pretraining cost. The paper also reports transfer experiments on detection and segmentation as well as image classification.

Read it for

Figure 1 and Section 3 explain the architecture; Figures 2–3 show reconstructions and Section 4.1 tests the design choices. You can skim the transfer setup in Section 5 until you want to compare downstream tasks.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.