G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models

2025/06/02 by Zhang, Tianjiao, Zhang, Fei, Yao, Jiangchao +2 · 4 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.01539

Abstract

This paper considers the problem of utilizing a large-scale text-to-image diffusion model to tackle the challenging Inexact Segmentation (IS) task. Unlike traditional approaches that rely heavily on discriminative-model-based paradigms or dense visual representations derived from internal attention mechanisms, our method focuses on the intrinsic generative priors in Stable Diffusion~(SD). Specifically, we exploit the pattern discrepancies between original images and mask-conditional generated images to facilitate a coarse-to-fine segmentation refinement by establishing a semantic correspondence alignment and updating the foreground probability. Comprehensive quantitative and qualitative experiments validate the effectiveness and superiority of our plug-and-play design, underscoring the potential of leveraging generation discrepancies to model dense representations and encouraging further exploration of generative approaches for solving discriminative tasks.

Citations

• Image Segmentation in Foundation Model Era: A Survey
• Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
• Relation Rectification in Diffusion Model
• Audio-Visual Segmentation via Unlabeled Frame Exploitation
• Sora as a World Model? A Complete Survey on Text-to-Video Generation
• SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
• UniGS: Unified Representation for Image Generation and Segmentation
• SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference
• Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation
• Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation
• Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
• AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
• Dense Text-to-Image Generation with Attention Modulation
• Disentangle then Parse:Night-time Semantic Segmentation with Illumination Disentanglement
• Exploiting Counter-Examples for Active Learning with Partial labels
• Multi-Modal Prototypes for Open-World Semantic Segmentation
• Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
• Monte Carlo Linear Clustering with Single-Point Supervision is Enough for Infrared Small Target Detection
• Micrograph segmentations for DDEVD
• Segment Anything
• WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentation
• Your Diffusion Model is Secretly a Zero-Shot Classifier
• DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery
• GPT-4 Technical Report
• Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
• Exploit CAM by itself: Complementary Learning System for Weakly Supervised Semantic Segmentation
• Unleashing Text-to-Image Diffusion Models for Visual Perception
• Token Contrast for Weakly-Supervised Semantic Segmentation
• Self Correspondence Distillation for End-to-End Weakly-Supervised Semantic Segmentation
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
• Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision
• Open-vocabulary Object Segmentation with Diffusion Models
• CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation
• Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs
• SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation
• Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors
• Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation
• LAION-5B: An open large-scale dataset for training next generation image-text models
• What the DAAM: Interpreting Stable Diffusion Using Cross Attention
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding
• ReCo: Retrieve and Co-segment for Zero-shot Transfer
• Video Diffusion Models
• Rethinking Semantic Segmentation: A Prototype View
• Multi-class Token Transformer for Weakly Supervised Semantic Segmentation
• GroupViT: Semantic Segmentation Emerges from Text Supervision
• Splicing ViT Features for Semantic Appearance Transfer
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Masked-attention Mask Transformer for Universal Image Segmentation
• Extract Free Dense Labels from CLIP
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling
• Complementary Patch for Weakly Supervised Semantic Segmentation
• Multi-scale Matching Networks for Semantic Correspondence
• SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers
• Unsupervised Part Segmentation through Disentangling Appearance and Shape
• DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
• Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Space-Time Correspondence as a Contrastive Random Walk
• Language Models are Few-Shot Learners
• CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
• Learning Feature Descriptors using Camera Pose Supervision
• Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
• Revisiting Pose-Normalization for Fine-Grained Few-Shot Recognition
• Deep Object Co-segmentation via Spatial-Semantic Network Modulation
• Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
• Learning Correspondence from the Cycle-Consistency of Time
• Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Learning Deep Features for Discriminative Localization
• Microsoft COCO: Common Objects in Context
• Distinctive Image Features from Scale-Invariant Keypoints
• Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network

Cited by

• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
• Decouple before Align: Visual Disentanglement Enhances Prompt Tuning
• MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition
• ConText: Driving In-context Learning for Text Removal and Segmentation

Related

• Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
• GS: Generative Segmentation via Label Diffusion
• Are Diffusion Models Vision-And-Language Reasoners?
• Segment Any-Quality Images with Generative Latent Space Enhancement