Personalize Segment Anything Model with One Shot

2023/05/04 by Renrui Zhang, Zhang, Renrui, Zhengkai Jiang +13 · 57 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2305.03048

openalex publication_date 2023/05/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models. Despite the generality, customizing SAM for specific visual concepts without man-powered prompting is under explored, e.g., automatically segmenting your pet dog in different images. In this paper, we propose a training-free Personalization approach for SAM, termed as PerSAM. Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior, and segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement. In this way, we effectively adapt SAM for private use without any training. To further alleviate the mask ambiguity, we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce two learnable weights for multi-scale masks, only training 2 parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for personalized evaluation, and test our methods on video object segmentation with competitive performance. Besides, our approach can also enhance DreamBooth to personalize Stable Diffusion for text-to-image generation, which discards the background disturbance for better target appearance learning. Code is released at https://github.com/ZrrSkywalker/Personalize-SAM

Cited by

• Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
• Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining
• Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting
• Memory-Enhanced SAM3 for Occlusion-Robust Surgical Instrument Segmentation
• Omni-Referring Image Segmentation
• Automated Annotation of Shearographic Measurements Enabling Weakly Supervised Defect Detection
• Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion
• Supervise Less, See More: Training-free Nuclear Instance Segmentation with Prototype-Guided Prompting
• SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM
• Beyond Visual Cues: Leveraging General Semantics as Support for Few-Shot Segmentation
• Unbiased Semantic Decoding with Vision Foundation Models for Few-shot Segmentation
• Bridging Granularity Gaps: Hierarchical Semantic Learning for Cross-domain Few-shot Segmentation
• STORM: Segment, Track, and Object Re-Localization from a Single Image
• Divide-and-Conquer Decoupled Network for Cross-Domain Few-Shot Segmentation
• MapSAM2: Adapting SAM2 for Automatic Segmentation of Historical Map Images and Time Series
• Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2
• Memory-SAM: Human-Prompt-Free Tongue Segmentation via Retrieval-to-Prompt
• Enhancing Zero-Shot Anomaly Detection: CLIP-SAM Collaboration with Cascaded Prompts
• Unified Open-World Segmentation with Multi-Modal Prompts
• Personalized Vision via Visual In-Context Learning
• Attack for Defense: Adversarial Agents for Point Prompt Optimization Empowering Segment Anything Model
• MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
• Towards PerSense++: Advancing Training-Free Personalized Instance Segmentation in Dense Images
• LENS: Learning to Segment Anything with Unified Reinforced Reasoning
• SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens
• TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object Segmentation and Tracking in Surgical Scenarios
• Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark
• Region-aware Depth Scale Adaptation with Sparse Measurements
• Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentation
• Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation
• Inter- and Intra-image Refinement for Few Shot Segmentation
• CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning
• No time to train! Training-Free Reference-Based Instance Segmentation
• Prompt learning with bounding box constraints for medical image segmentation
• ViRefSAM: Visual Reference-Guided Segment Anything Model for Remote Sensing Segmentation
• ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts
• Few-Shot Segmentation of Historical Maps via Linear Probing of Vision Foundation Models
• Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
• SynPo: Boosting Training-Free Few-Shot Medical Segmentation via High-Quality Negative Prompts
• Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment
• Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation
• Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT
• SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
• Evaluating war-induced damage to agricultural land in the Gaza Strip since October 2023 using PlanetScope and SkySat imagery
• Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment
• Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding
• UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
• Causal Prompt Calibration Guided Segment Anything Model for Open-Vocabulary Multi-Entity Segmentation
• INSID3: Training-Free In-Context Segmentation with DINOv3
• Free-Lunch Augmentation by Revisiting Diffusion-Based Data Generation for Cross-Domain Few-Shot Object Detection
• S⁴M: Boosting Semi-Supervised Instance Segmentation with SAM
• UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation
• GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoning
• DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency
• Parameter-Free Fine-tuning via Redundancy Elimination for Vision Foundation Models
• Detect Anything 3D in the Wild
• MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation
• Studying Image Diffusion Features for Zero-Shot Video Object Segmentation
• CMaP-SAM: Contraction Mapping Prior for SAM-driven Few-shot Segmentation

Related

• Segment Anything in High Quality
• Micrograph segmentations for DDEVD
• Stable Segment Anything Model
• Segment Anything
• Segment Anything without Supervision