Segment Anything

Inside the paper

The problem

Segmentation systems usually depended on task-specific training data, and collecting masks at a large scale was expensive. The authors wanted a model that could respond to different cues about what part of an image to segment.

The idea

SAM predicts masks from an image and a prompt such as a point or box. A model-assisted annotation process supplies data to train the model and then uses improved models to help collect more masks.

How it works

• An image encoder computes an image representation that can be reused for multiple prompts on the same image.
• A prompt encoder represents the user’s cue, and a lightweight decoder combines it with the image representation to predict masks.
• The model can produce multiple candidate masks for an ambiguous prompt, such as a point that could refer to a part or a whole object.
• The data engine progresses from assisted manual annotation to automatic mask generation, using the collected masks to improve the model.

What it showed

The paper evaluates zero-shot transfer to several segmentation-related tasks and compares its masks with human annotations and prior systems. It also describes the resulting dataset and examines its distribution and mask quality.

Read it for

Figure 1 and Sections 2–4 explain the task, model, and data collection loop; Section 7 shows the transfer evaluations. You can skim the dataset details in Section 5 initially and return when assessing coverage.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.