What Is Segment Anything Model (SAM)? Segmentation Principles and Uses
The encoder-decoder architecture and prompt-based approach behind Meta AI's general-purpose segmentation model SAM, and how it evolved into SAM 2 and SAM 3.
SAM (Segment Anything Model) is a general-purpose segmentation model from Meta AI Research that, given a simple prompt like a point or a box, finds an object’s pixel-level boundary (mask) in an image without needing to have been trained on that specific class. As covered in SAM vs YOLO, SAM is built to solve “exactly where does this object end,” rather than “what is this object.”
SAM’s architecture: encoder + prompt encoder + decoder
SAM is made up of three parts.
- Image encoder — processes the entire input image once to extract features; this is the relatively heavy computation.
- Prompt encoder — encodes user input (a point, box, or text prompt) into a form the model can understand.
- Mask decoder — a lightweight decoder that combines the image features and prompt information to quickly produce a segmentation mask.
Thanks to this architecture, the heavy image encoding only needs to run once — after that, different prompts can be tried by re-running only the lightweight mask decoder, segmenting a variety of objects quickly.
SA-1B: a large-scale segmentation dataset
SAM was trained on the SA-1B dataset, made up of 11 million diverse, high-resolution images and 1.1 billion segmentation masks. Training on data this large and diverse gives it zero-shot performance across a wide range of objects, without needing fine-tuning for any specific class.
From SAM to SAM 2 to SAM 3
Through SAM 2 (often written SAM2) and SAM 3 (SAM3), its capabilities expanded as follows.
| Version | Core capability |
|---|---|
| SAM | Segments a single object in a still image via point/box prompts |
| SAM 2 | Segments and tracks objects not just in still images but in video |
| SAM 3 | Given a “concept” specified via text or image examples, segments and tracks every matching instance across images and video |
For more on SAM 3, see Meta Unveils SAM 3, a Concept-Based Segmentation Model.
Model size and practical use
Per Ultralytics’ documentation, the SAM base model (sam_b.pt) is about 375MB with roughly 93.7 million parameters. The SAM family is used for data labeling that needs precise segmentation masks, image/video editing tools, and industrial inspection requiring precise region extraction. Since SAM itself doesn’t classify “what” an object is, it needs to be paired with a detection model like YOLO or a separate classifier when class recognition is required.
Conclusion
SAM is a general-purpose segmentation model that, through its image encoder, prompt encoder, and mask decoder architecture, accurately segments a wide range of objects without class-specific training. Its capabilities keep expanding — from still images, to video tracking (SAM 2), to concept-based multi-instance segmentation (SAM 3).