Few-Shot Segmentation via Cycle-Consistent Transformer

2021/06/04 by Gengwei Zhang, Zhang, Gengwei, Guoliang Kang +5 · 11 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2106.02320

openalex publication_date 2021/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Cycle-Consistent TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-5ⁱ and COCO-20ⁱ datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art methods by 5.6% and 7.1% respectively.

Citations

• The Pascal Visual Object Classes (VOC) Challenge
• ImageNet Large Scale Visual Recognition Challenge
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Hypercorrelation Squeeze for Few-Shot Segmentation

Cited by

• DistillFSS: Synthesizing Few-Shot Knowledge into a Lightweight Segmentation Model
• Matching-Based Few-Shot Semantic Segmentation Models Are Interpretable by Design
• Data Factory with Minimal Human Effort Using VLMs
• FS-SAM2: Adapting Segment Anything Model 2 for Few-Shot Semantic Segmentation via Low-Rank Adaptation
• Object-level Correlation for Few-Shot Segmentation
• SPENet: Self-guided Prototype Enhancement Network for Few-shot Medical Image Segmentation
• Cross-Domain Few-Shot Segmentation via Ordinary Differential Equations over Time Intervals
• Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild
• Textual and Visual Guided Task Adaptation for Source-Free Cross-Domain Few-Shot Segmentation
• MOVE: Motion-Guided Few-Shot Video Object Segmentation
• Balancing Conservatism and Aggressiveness: Prototype-Affinity Hybrid Network for Few-Shot Segmentation

Related

• FCC: Fully Connected Correlation for Few-Shot Segmentation
• Elimination of Non-Novel Segments at Multi-Scale for Few-Shot Segmentation
• Feature-Proxy Transformer for Few-Shot Segmentation
• Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification
• Few-Shot Segmentation with Global and Local Contrastive Learning