PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications

2025/06/23 by Bonazzi, Pietro, Farronato, Nicola, Zihlmann, Stefan +2 · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.18807

Abstract

Real-time, on-device segmentation is critical for latency-sensitive and privacy-aware applications like smart glasses and IoT devices. We introduce PicoSAM2, a lightweight (1.3M parameters, 336M MACs) promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500. It builds on a depthwise separable U-Net, with knowledge distillation and fixed-point prompt encoding to learn from the Segment Anything Model 2 (SAM2). On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively. The quantized model (1.22MB) runs at 14.3 ms on the IMX500-achieving 86 MACs/cycle, making it the only model meeting both memory and compute constraints for in-sensor deployment. Distillation boosts LVIS performance by +3.5% mIoU and +5.1% mAP. These results demonstrate that efficient, promptable segmentation is feasible directly on-camera, enabling privacy-preserving vision without cloud or host processing.

Citations

• Data Generation for Hardware-Friendly Post-Training Quantization
• Lite-SAM Is Actually What You Need for Segment Everything
• TinySAM: Pushing the Envelope for Efficient Segment Anything Model
• MobileSAMv2: Faster Segment Anything to Everything
• Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO
• EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization
• Fast Segment Anything
• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
• Micrograph segmentations for DDEVD
• Segment Anything
• HPTQ: Hardware-Friendly Post Training Quantization
• A Survey of Quantization Methods for Efficient Neural Network Inference
• MCUNet: Tiny Deep Learning on IoT Devices
• Up or Down? Adaptive Rounding for Post-Training Quantization
• FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• LVIS: A Dataset for Large Vocabulary Instance Segmentation
• Xception: Deep Learning with Depthwise Separable Convolutions
• V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Distilling the Knowledge in a Neural Network
• Microsoft COCO: Common Objects in Context

Cited by

• Promptable Fire Segmentation: Unleashing SAM2's Potential for Real-Time Mobile Deployment with Strategic Bounding Box Guidance

Related

• EdgeTAM: On-Device Track Anything Model
• RepViT-SAM: Towards Real-Time Segmenting Anything
• Tiny-YOLOSAM: Fast Hybrid Image Segmentation
• Q-Segment: Segmenting Images In-Sensor for Vessel-Based Medical Diagnosis
• EdgeSAM: Prompt-In-the-Loop Distillation for SAM