text #4,

text #5score

text #6mask decoder

text #1image

text #2encoder

text #7,

text #8score

text #9conv

text #10prompt encoder

text #3image

text #11image

text #12,

text #13score

text #14embedding mask

text #15points

text #16box

text #17text

text #18valid masks

text #19Figure 4: Segment Anything Model (SAM) overview. A heavyweight image encoder outputs an image embedding that can

text #20then be efficiently queried by a variety of input prompts to produce object masks at amortized real-time speed. For ambiguous

text #21prompts corresponding to more than one object, SAM can output multiple valid masks and associated confidence scores.

text #223. Segment Anything Model

text #61loss [15, 45, 64] over masks. To rank masks, the model pre-

text #62dicts a confidence score (i.e., estimated IoU) for each mask.

text #23We next describe the Segment Anything Model (SAM)

text #63Efficiency. The overall model design is largely motivated

text #24for promptable segmentation. SAM has three components,

text #64by efficiency. Given a precomputed image embedding, the

text #25illustrated in Fig. 4: an image encoder, a flexible prompt

text #65prompt encoder and mask decoder run in a web browser, on

text #26encoder, and a fast mask decoder. We build on Transformer

text #66CPU, in ∼50ms. This runtime performance enables seam-

text #27vision models [14, 33, 20, 62] with specific tradeoffs for

text #67less, real-time interactive prompting of our model.

text #28(amortized) real-time performance. We describe these com-

text #29ponents at a high-level here, with details in §A.

text #68Losses and training. We supervise mask prediction with

text #69the linear combination of focal loss [65] and dice loss [73]

text #30Image encoder. Motivated by scalability and powerful pre-

text #70used in [14]. We train for the promptable segmentation task

text #31training methods, we use an MAE [47] pre-trained Vision

text #71using a mixture of geometric prompts (for text prompts see

text #32Transformer (ViT) [33] minimally adapted to process high

text #72§7.5). Following [92, 37], we simulate an interactive setup

text #33resolution inputs [62]. The image encoder runs once per

text #73by randomly sampling prompts in 11 rounds per mask, al-

text #34image and can be applied prior to prompting the model.

text #74lowing SAM to integrate seamlessly into our data engine.

text #35Prompt encoder. We consider two sets of prompts: sparse

text #754. Segment Anything Data Engine

text #36(points, boxes, text) and dense (masks).

text #37We represent

text #38points and boxes by positional encodings [95] summed with

text #76As segmentation masks are not abundant on the inter-

text #39learned embeddings for each prompt type and free-form text

text #77net, we built a data engine to enable the collection of our

text #40with an off-the-shelf text encoder from CLIP [82]. Dense

text #781.1B mask dataset, SA-1B.

text #79The data engine has three

text #41prompts (i.e., masks) are embedded using convolutions and

text #80stages: (1) a model-assisted manual annotation stage, (2) a

text #42summed element-wise with the image embedding.

text #81semi-automatic stage with a mix of automatically predicted

text #43Mask decoder. The mask decoder efficiently maps the im-

text #82masks and model-assisted annotation, and (3) a fully auto-

text #44age embedding, prompt embeddings, and an output token

text #83matic stage in which our model generates masks without

text #45to a mask. This design, inspired by [14, 20], employs a

text #84annotator input. We go into details of each next.

text #46modification of a Transformer decoder block [103] followed

text #85Assisted-manual stage. In the first stage, resembling clas-

text #47by a dynamic mask prediction head. Our modified decoder

text #86sic interactive segmentation, a team of professional annota-

text #48block uses prompt self-attention and cross-attention in two

text #87tors labeled masks by clicking foreground / background ob-

text #49directions (prompt-to-image embedding and vice-versa) to

text #88ject points using a browser-based interactive segmentation

text #50update all embeddings. After running two blocks, we up-

text #89tool powered by SAM. Masks could be refined using pixel-

text #51sample the image embedding and an MLP maps the output

text #90precise “brush” and “eraser” tools. Our model-assisted an-

text #52token to a dynamic linear classifier, which then computes

text #91notation runs in real-time directly inside a browser (using

text #53the mask foreground probability at each image location.

text #92precomputed image embeddings) enabling a truly interac-

text #54Resolving ambiguity. With one output, the model will av-

text #93tive experience. We did not impose semantic constraints for

text #55erage multiple valid masks if given an ambiguous prompt.

text #94labeling objects, and annotators freely labeled both “stuff”

text #56To address this, we modify the model to predict multiple

text #95and “things” [1].

text #96We suggested annotators label objects

text #57output masks for a single prompt (see Fig. 3). We found

text #97they could name or describe, but did not collect these names

text #583 mask outputs is sufficient to address most common cases

text #98or descriptions. Annotators were asked to label objects in

text #59(nested masks are often at most three deep: whole, part, and

text #99order of prominence and were encouraged to proceed to the

text #60subpart). During training, we backprop only the minimum

text #100next image once a mask took over 30 seconds to annotate.

text #1015

image #102

image #103

image #104

image #105

shape #106

shape #107

shape #108

shape #109

shape #110

shape #111

shape #112

shape #113

shape #114

shape #115

shape #116

shape #117

shape #118

shape #119

shape #120

shape #121

shape #122

shape #123

shape #124

shape #125

shape #126

shape #127

shape #128

shape #129

shape #130

shape #131

shape #132

shape #133

shape #134

shape #135

shape #136

shape #137

shape #138

shape #139

shape #140

shape #141

shape #142

shape #143

shape #144

shape #145

shape #146

shape #147

shape #148

shape #149

shape #150