SegDiff: Image Segmentation with Diffusion Probabilistic Models

2021/12/01 by Amit, Tomer, Shaharbany, Tal, Nachmani, Eliya +1 · 24 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2112.00390

Abstract

Diffusion Probabilistic Methods are employed for state-of-the-art image generation. In this work, we present a method for extending such models for performing image segmentation. The method learns end-to-end, without relying on a pre-trained backbone. The information in the input image and in the current estimation of the segmentation map is merged by summing the output of two encoders. Additional encoding layers and a decoder are then used to iteratively refine the segmentation map, using a diffusion model. Since the diffusion model is probabilistic, it is applied multiple times, and the results are merged into a final segmentation map. The new method produces state-of-the-art results on the Cityscapes validation set, the Vaihingen building segmentation benchmark, and the MoNuSeg dataset.

Cited by

• CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Models
• LatentFM: A Latent Flow Matching Approach for Generative Medical Image Segmentation
• Diffusion Model in Latent Space for Medical Image Segmentation Task
• MedCondDiff: Lightweight, Robust, Semantically Guided Diffusion for Medical Image Segmentation
• From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition
• DreamPose3D: Hallucinative Diffusion with Prompt Learning for 3D Human Pose Estimation
• LaneDiffusion: Improving Centerline Graph Learning via Prior Injected BEV Feature Generation
• TA-LSDiff:Topology-Aware Diffusion Guided by a Level Set Energy for Pancreas Segmentation
• WaterFlow: Explicit Physics-Prior Rectified Flow for Underwater Saliency Mask Generation
• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
• Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation
• Bézier Meets Diffusion: Robust Generation Across Domains for Medical Image Segmentation
• High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling
• DUDE: Diffusion-Based Unsupervised Cross-Domain Image Retrieval
• Q-Sched: Pushing the Boundaries of Few-Step Diffusion Models with Quantization-Aware Scheduling
• SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3
• Generic Event Boundary Detection via Denoising Diffusion
• Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
• SDMatte: Grafting Diffusion Models for Interactive Matting
• ReCoSeg++:Extended Residual-Guided Cross-Modal Diffusion for Brain Tumor Segmentation
• Flow Stochastic Segmentation Networks
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model
• A Comprehensive Review of Diffusion Models in Smart Agriculture: Progress, Applications, and Challenges
• Diffusion-FS: Multimodal Free-Space Prediction via Diffusion for Autonomous Driving

Related

• Towards Better Certified Segmentation via Diffusion Models
• Diffusion Models for Implicit Image Segmentation Ensembles
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model
• Label-Efficient Semantic Segmentation with Diffusion Models
• Analysing Diffusion Segmentation for Medical Images