SegDiff: Image Segmentation with Diffusion Probabilistic Models

2021/12/01 by Tomer Amit, Tal Shaharbany, Amit, Tomer +5 · 44 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Artificial intelligence #Benchmark (surveying) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Encoder #Encoding (memory) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Geography #Image (mathematics) #Image segmentation #Machine Learning (cs.LG) #Pattern recognition (psychology) #Probabilistic logic #Scale-space segmentation #Segmentation #Segmentation-based object categorization #Statistical model #Video Surveillance and Tracking Methods #cs.AI #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2112.00390

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/12/01 · arxiv created 2022/09/07 · arxiv updated 2022/09/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Diffusion Probabilistic Methods are employed for state-of-the-art image generation. In this work, we present a method for extending such models for performing image segmentation. The method learns end-to-end, without relying on a pre-trained backbone. The information in the input image and in the current estimation of the segmentation map is merged by summing the output of two encoders. Additional encoding layers and a decoder are then used to iteratively refine the segmentation map, using a diffusion model. Since the diffusion model is probabilistic, it is applied multiple times, and the results are merged into a final segmentation map. The new method produces state-of-the-art results on the Cityscapes validation set, the Vaihingen building segmentation benchmark, and the MoNuSeg dataset.

Citations

• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Adversarial Examples for Semantic Image Segmentation
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Denoising Diffusion Probabilistic Models
• Improved Denoising Diffusion Probabilistic Models
• SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• Diffusion Models Beat GANs on Image Synthesis
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• Variational Diffusion Models
• Structured Denoising Diffusion Models in Discrete State-Spaces

Cited by

• CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Models
• LatentFM: A Latent Flow Matching Approach for Generative Medical Image Segmentation
• Diffusion Model in Latent Space for Medical Image Segmentation Task
• MedCondDiff: Lightweight, Robust, Semantically Guided Diffusion for Medical Image Segmentation
• From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition
• DreamPose3D: Hallucinative Diffusion with Prompt Learning for 3D Human Pose Estimation
• LaneDiffusion: Improving Centerline Graph Learning via Prior Injected BEV Feature Generation
• TA-LSDiff:Topology-Aware Diffusion Guided by a Level Set Energy for Pancreas Segmentation
• WaterFlow: Explicit Physics-Prior Rectified Flow for Underwater Saliency Mask Generation
• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
• Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation
• Bézier Meets Diffusion: Robust Generation Across Domains for Medical Image Segmentation
• High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
• DUDE: Diffusion-Based Unsupervised Cross-Domain Image Retrieval
• Q-Sched: Pushing the Boundaries of Few-Step Diffusion Models with Quantization-Aware Scheduling
• SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3
• Generic Event Boundary Detection via Denoising Diffusion
• Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
• SDMatte: Grafting Diffusion Models for Interactive Matting
• ReCoSeg++:Extended Residual-Guided Cross-Modal Diffusion for Brain Tumor Segmentation
• Flow Stochastic Segmentation Networks
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model
• A Comprehensive Review of Diffusion Models in Smart Agriculture: Progress, Applications, and Challenges
• Diffusion-FS: Multimodal Free-Space Prediction via Diffusion for Autonomous Driving
• Diffusion Models for Solving Inverse Problems via Posterior Sampling with Piecewise Guidance
• CPKD: Clinical Prior Knowledge-Constrained Diffusion Models for Surgical Phase Recognition in Endoscopic Submucosal Dissection
• LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion
• DMCIE: Diffusion Model with Concatenation of Inputs and Errors to Improve the Accuracy of the Segmentation of Brain Tumors in MRI Images
• FreeDNA: Endowing Domain Adaptation of Diffusion-Based Dense Prediction with Training-Free Domain Noise Alignment
• Echo-DND: A dual noise diffusion model for robust and precise left ventricle segmentation in echocardiography
• Symmetrical Flow Matching: Unified Image Generation, Segmentation, and Classification with Score-Based Generative Models
• A Data-Driven Diffusion-based Approach for Audio Deepfake Explanations
• G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models
• Compositional Scene Understanding through Inverse Generative Modeling
• On Denoising Walking Videos for Gait Recognition
• Recent Advances in Diffusion Models for Hyperspectral Image Processing and Analysis: A Review
• FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization
• Computationally Efficient Diffusion Models in Medical Imaging: A Comprehensive Review
• LDPoly: Latent Diffusion for Polygonal Road Outline Extraction in Large-Scale Topographic Mapping
• DIVE: Inverting Conditional Diffusion Models for Discriminative Tasks
• DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation
• SD-ReID: View-aware Stable Diffusion for Aerial-Ground Person Re-Identification
• PathSegDiff: Pathology Segmentation using Diffusion model representations
• Diffusion Based Ambiguous Image Segmentation
• Studying Image Diffusion Features for Zero-Shot Video Object Segmentation

Related

• Towards Better Certified Segmentation via Diffusion Models
• Diffusion Models for Implicit Image Segmentation Ensembles
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model
• Label-Efficient Semantic Segmentation with Diffusion Models
• Analysing Diffusion Segmentation for Medical Images