mlx-community/sam3-image

🎯 MLX SAM3

Segment Anything Model 3 — Native Apple Silicon Implementation

This is an MLX port of Meta's SAM3 model, optimized for native execution on Apple Silicon (M1/M2/M3/M4) Macs.

📖 Learn more: Check out the accompanying blog post explaining the SAM3 architecture and this implementation.

Model Description

SAM3 (Segment Anything Model 3) is a powerful image segmentation model that can segment objects in images using:

• Text prompts — Describe what you want to segment ("car", "person", "dog")
• Box prompts — Draw bounding boxes to include or exclude regions

This MLX port provides native Apple Silicon performance, leveraging Apple's MLX framework for optimized inference on Mac.

Intended Uses

• Interactive image segmentation on Apple Silicon Macs
• Object detection and masking with text descriptions
• Region-based segmentation using bounding box prompts
• Rapid prototyping of segmentation workflows on Mac

How to Use

Installation

# Clone the repository
git clone https://github.com/Deekshith-Dade/mlx_sam3.git
cd mlx-sam3

# Install with uv (recommended)
uv sync

# Or with pip
pip install -e .

Python API

from PIL import Image
from sam3 import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# Load model (auto-downloads weights on first run)
model = build_sam3_image_model()
processor = Sam3Processor(model, confidence_threshold=0.5)

# Load and process an image
image = Image.open("your_image.jpg")
state = processor.set_image(image)

# Segment with text prompt
state = processor.set_text_prompt("person", state)

# Access results
masks = state["masks"]       # Binary segmentation masks
boxes = state["boxes"]       # Bounding boxes [x0, y0, x1, y1]
scores = state["scores"]     # Confidence scores

print(f"Found {len(scores)} objects")

Web Interface

Launch the interactive web application:

cd app && ./run.sh

• Frontend: http://localhost:3000
• API: http://localhost:8000
• API Docs: http://localhost:8000/docs

Requirements

Requirement	Version	Notes
macOS	13.0+	Apple Silicon required (M1/M2/M3/M4)
Python	3.13+	Required for MLX compatibility
Node.js	18+	For the web interface (optional)

⚠️ Apple Silicon Only: This implementation uses MLX, which is optimized exclusively for Apple Silicon.

Model Details

• Architecture: SAM3 with ViTDet backbone
• Framework: MLX (Apple's machine learning framework)
• Weights: Converted from original PyTorch weights
• Model Size: ~3.5GB

Limitations

• Runs only on Apple Silicon Macs (M1/M2/M3/M4)
• Requires macOS 13.0 or later
• Python 3.13+ required for MLX compatibility

Citation

If you use this model, please cite the original SAM3 paper and this MLX implementation:

@misc{mlx-sam3,
  author = {Deekshith Dade},
  title = {MLX SAM3: Native Apple Silicon Implementation},
  year = {2024},
  url = {https://github.com/Deekshith-Dade/mlx_sam3}
}

Links

• GitHub Repository: https://github.com/Deekshith-Dade/mlx_sam3
• Blog Post: https://deekshith.me/blog/mlx-sam3
• Original SAM3: https://huggingface.co/facebook/sam3

Acknowledgments

• Meta AI for the original SAM3 model
• Apple MLX Team for the MLX framework
• The open-source community for continuous inspiration

---

Built with ❤️ for Apple Silicon