SAM 2 is Meta's Segment Anything Model 2, a foundation model for promptable visual segmentation in both images and videos. The repository provides inference code, trained model checkpoints, and example notebooks demonstrating how to use the model. The primary implementation language is Jupyter Notebook, reflecting the emphasis on accessible, example-driven documentation.
The model extends the original SAM to handle video by treating images as single-frame videos. Its architecture uses a simple transformer design with streaming memory to enable real-time video processing. The model was trained on the SA-V dataset, which Meta describes as the largest video segmentation dataset to date, collected through a model-in-the-loop data engine that improves both the model and data through user interaction.
SAM 2 provides two main prediction interfaces. The SAM2ImagePredictor class handles static image segmentation with an API closely resembling the original SAM, supporting both interactive prompting and automatic mask generation. For video, the SAM2VideoPredictor enables promptable segmentation and tracking across multiple objects, maintaining an inference state to track interactions throughout a video. As of December 2024, the video predictor was updated to support independent per-object inference and multi-object tracking without requiring all objects to be prompted simultaneously.
The repository includes four model checkpoint variants in the SAM 2.1 release from September 2024. These range from the sam2.1_hiera_tiny at 38.9 million parameters running at 91.2 FPS, to the sam2.1_hiera_large at 224.4 million parameters running at 39.5 FPS. Performance metrics are provided across multiple benchmarks including the SA-V test set, MOSE validation set, and LVOS v2.
Installation requires Python 3.10 or higher, PyTorch 2.5.1 or later, and TorchVision 0.20.1 or later. The setup process involves compiling a custom CUDA kernel, though the repository notes that SAM 2 can still function if this compilation fails, with only some post-processing functionality potentially limited. Models can be downloaded individually or via a batch download script, and can alternatively be loaded from Hugging Face.
The repository shares contributors with pytorch/pytorch, huggingface/transformers, and flutter/flutter, indicating cross-project collaboration within the broader machine learning ecosystem.
Recent updates include support for torch.compile on the entire SAM 2 model for videos, activated via the vos_optimized parameter, which provides major speedup for video object segmentation inference. The repository also released training and fine-tuning code in September 2024, along with frontend and backend code for the SAM 2 web demo, expanding beyond inference-only capabilities to enable model customization and deployment.
