Adaptively Distilled ControlNet: Accelerated Training and Superior Sampling for Medical Image Synthesis

2025/07/31 by Kunpeng Qiu, Qiu, Kunpeng, Zhiying Zhou +3 · 2 citations
Computer Science · #AI in cancer detection #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Medical Image Segmentation Techniques

paper · pdf · doi:10.48550/arxiv.2507.23652

openalex publication_date 2025/07/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Medical image annotation is constrained by privacy concerns and labor-intensive labeling, significantly limiting the performance and generalization of segmentation models. While mask-controllable diffusion models excel in synthesis, they struggle with precise lesion-mask alignment. We propose Adaptively Distilled ControlNet, a task-agnostic framework that accelerates training and optimization through dual-model distillation. Specifically, during training, a teacher model, conditioned on mask-image pairs, regularizes a mask-only student model via predicted noise alignment in parameter space, further enhanced by adaptive regularization based on lesion-background ratios. During sampling, only the student model is used, enabling privacy-preserving medical image generation. Comprehensive evaluations on two distinct medical datasets demonstrate state-of-the-art performance: TransUNet improves mDice/mIoU by 2.4%/4.2% on KiTS19, while SANet achieves 2.6%/3.5% gains on Polyps, highlighting its effectiveness and superiority. Code is available at GitHub.

Citations

• SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model
• Rethinking Brain Tumor Segmentation from the Frequency Domain Perspective
• Noise-Consistent Siamese-Diffusion for Medical Image Synthesis and Segmentation
• Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion
• Enhancing Fetal Plane Classification Accuracy with Data Augmentation Using Diffusion Models
• Learn From Zoom: Decoupled Supervised Contrastive Learning For WCE Image Classification
• AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model
• ArSDM: Colonoscopy Images Synthesis with Adaptive Refinement Semantic Diffusion Models
• Consistency Models
• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• On Distillation of Guided Diffusion Models
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Classifier-Free Diffusion Guidance
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Shallow Attention Network for Polyp Segmentation
• Diffusion Models Beat GANs on Image Synthesis
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Kvasir-SEG: A Segmented Polyp Dataset
• The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes
• Neural Discrete Representation Learning
• A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images
• Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information

Cited by

• vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs
• PairUni: Pairwise Training for Unified Multimodal Language Models

Related

• MedDiff-FT: Data-Efficient Diffusion Model Fine-tuning with Structural Guidance for Controllable Medical Image Synthesis
• MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairs
• DiffBoost: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model
• Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models
• CS²: A Controllable and Simultaneous Synthesizer of Images and Annotations with Minimal Human Intervention