No time to train! Training-Free Reference-Based Instance Segmentation

2025/07/03 by Miguel Espinosa, Chenhongyi Yang, Espinosa, Miguel +7 · 1 citation
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2507.02798

openalex publication_date 2025/07/03 · openalex created_date 2025/10/20 · openalex updated_date 2026/08/02

Abstract

The performance of image segmentation models has historically been constrained by the high cost of collecting large-scale annotated data. The Segment Anything Model (SAM) alleviates this original problem through a promptable, semantics-agnostic, segmentation paradigm and yet still requires manual visual-prompts or complex domain-dependent prompt-generation rules to process a new image. Towards reducing this new burden, our work investigates the task of object segmentation when provided with, alternatively, only a small set of reference images. Our key insight is to leverage strong semantic priors, as learned by foundation models, to identify corresponding regions between a reference and a target image. We find that correspondences enable automatic generation of instance-level segmentation masks for downstream tasks and instantiate our ideas via a multi-stage, training-free method incorporating (1) memory bank construction; (2) representation aggregation and (3) semantic-aware feature matching. Our experiments show significant improvements on segmentation metrics, leading to state-of-the-art performance on COCO FSOD (36.8% nAP), PASCAL VOC Few-Shot (71.2% nAP50) and outperforming existing training-free approaches on the Cross-Domain FSOD benchmark (22.4% nAP).

Citations

• LightSAM: Parameter-Agnostic Sharpness-Aware Minimization
• Perception Encoder: The best visual embeddings are not at the output of the network
• There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Tasks
• A Simple Image Segmentation Framework via In-Context Examples
• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
• Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation
• SAM 2: Segment Anything in Images and Videos
• Investigating the Segment Anything Foundation Model for Mapping Smallholder Agriculture Field Boundaries Without Training Labels
• T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy
• VRP-SAM: SAM with Visual Reference Prompt
• Cross-Domain Few-Shot Object Detection via Enhanced Open-Set Object Detector
• Learning to Prompt Segment Anything Models
• Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions
• Tokenize Anything via Prompting
• Boosting Segment Anything Model Towards Open-Vocabulary Learning
• SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints
• Detect Everything with Few Examples
• Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations
• LISA: Reasoning Segmentation via Large Language Model
• Semantic-SAM: Segment and Recognize Anything at Any Granularity
• The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot
• Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
• Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
• Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey
• Personalize Segment Anything Model with One Shot
• What does CLIP know about a red circle? Visual prompt engineering for VLMs
• Segment Everything Everywhere All at Once
• SegGPT: Segmenting Everything In Context
• Micrograph segmentations for DDEVD
• Segment Anything
• Hierarchical Dense Correlation Distillation for Few-Shot Segmentation
• DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• NIFF: Alleviating Forgetting in Generalized Few-Shot Object Detection via Neural Instance Feature Forging
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning
• CD-FSOD: A Benchmark for Cross-domain Few-shot Object Detection
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation
• Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation
• iFS-RCNN: An Incremental Few-shot Instance Segmenter
• Self-Supervised Learning of Object Parts for Semantic Segmentation
• Exploring Plain Vision Transformer Backbones for Object Detection
• Few-Shot Object Detection with Fully Cross-Transformer
• Visual Prompt Tuning
• Learning What Not to Segment: A New Perspective on Few-Shot Segmentation
• Detecting Twenty-thousand Classes using Image-level Supervision
• Label, Verify, Correct: A Simple Few Shot Object Detection Method
• Masked-attention Mask Transformer for Universal Image Segmentation
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Scaling up instance annotation via label propagation
• Query Adaptive Few-Shot Object Detection with Heterogeneous Graph Convolutional Networks
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection
• Few-Shot Segmentation via Cycle-Consistent Transformer
• Generalized Few-Shot Object Detection without Forgetting
• Incremental Few-Shot Instance Segmentation
• Emerging Properties in Self-Supervised Vision Transformers
• Meta Faster R-CNN: Towards Accurate Few-Shot Object Detection with Attentive Feature Alignment
• Hypercorrelation Squeeze for Few-Shot Segmentation
• FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding
• Learning Transferable Visual Models From Natural Language Supervision
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• End-to-End Object Detection with Transformers
• Frustratingly Simple Few-Shot Object Detection
• Feature Weighting and Boosting for Few-Shot Segmentation
• Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning
• PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment
• Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector
• LVIS: A Dataset for Large Vocabulary Instance Segmentation
• TensorMask: A Foundation for Dense Object Segmentation
• Large-scale interactive object segmentation with human annotators
• Few-shot Object Detection via Feature Reweighting
• Mask R-CNN
• Fully Convolutional Instance-aware Semantic Segmentation
• V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
• Microsoft COCO: Common Objects in Context
• MedLSAM: Localize and Segment Anything Model for 3D CT Images

Cited by

• INSID3: Training-Free In-Context Segmentation with DINOv3

Related

• Synthetic Instance Segmentation from Semantic Image Segmentation Masks
• Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals
• Semantic-aware SAM for Point-Prompted Instance Segmentation
• SOS: Segment Object System for Open-World Instance Segmentation With Object Priors
• FreeSOLO: Learning to Segment Objects without Annotations