Dual Prompting Image Restoration with Diffusion Transformers

2025/04/24 by Dehong Kong, Kong, Dehong, Fan Li +11 · 30 citations
Computer Science · #Advanced Image Processing Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Enhancement Techniques

paper · pdf · doi:10.48550/arxiv.2504.17825

openalex publication_date 2025/04/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent state-of-the-art image restoration methods mostly adopt latent diffusion models with U-Net backbones, yet still facing challenges in achieving high-quality restoration due to their limited capabilities. Diffusion transformers (DiTs), like SD3, are emerging as a promising alternative because of their better quality with scalability. In this paper, we introduce DPIR (Dual Prompting Image Restoration), a novel image restoration method that effectivly extracts conditional information of low-quality images from multiple perspectives. Specifically, DPIR consits of two branches: a low-quality image conditioning branch and a dual prompting control branch. The first branch utilizes a lightweight module to incorporate image priors into the DiT with high efficiency. More importantly, we believe that in image restoration, textual description alone cannot fully capture its rich visual characteristics. Therefore, a dual prompting module is designed to provide DiT with additional visual cues, capturing both global context and local appearance. The extracted global-local visual prompts as extra conditional control, alongside textual prompts to form dual prompts, greatly enhance the quality of the restoration. Extensive experimental results demonstrate that DPIR delivers superior image restoration performance.

Citations

• 3D Priors-Guided Diffusion for Blind Face Restoration
• One Step Diffusion-based Super-Resolution with Time-Aware Distillation
• ControlNeXt: Powerful and Efficient Control for Image and Video Generation
• One-Step Effective Diffusion Network for Real-World Image Super-Resolution
• AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillation
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
• SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution
• SinSR: Diffusion-Based Image Super-Resolution in a Single Step
• Controlling Vision-Language Models for Multi-Task Image Restoration
• DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
• Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization
• ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Exploiting Diffusion Prior for Real-World Image Super-Resolution
• Visual Instruction Tuning
• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Scalable Diffusion Models with Transformers
• Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model
• Flow Matching for Generative Modeling
• Accurate Image Restoration with Attention Retractable Transformer
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Exploring CLIP for Assessing the Look and Feel of Images
• Activating More Pixels in Image Super-Resolution Transformer
• MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment
• Denoising Diffusion Restoration Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• SwinIR: Image Restoration Using Swin Transformer
• SwinIR: Image Restoration Using Swin Transformer
• MUSIQ: Multi-scale Image Quality Transformer
• Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
• SNIPS: Solving Noisy Inverse Problems Stochastically
• Diffusion Models Beat GANs on Image Synthesis
• Designing a Practical Degradation Model for Deep Blind Image\n Super-Resolution
• Learning Transferable Visual Models From Natural Language Supervision
• Taming Transformers for High-Resolution Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Denoising Diffusion Implicit Models
• Component Divide-and-Conquer for Real-World Image Super-Resolution
• PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
• Denoising Diffusion Probabilistic Models
• Neural Sparse Representation for Image Restoration
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Enhanced Deep Residual Networks for Single Image Super-Resolution
• Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deep Learning Face Attributes in the Wild
• Making a “Completely Blind” Image Quality Analyzer

Cited by

• FlowSteer: Conditioning Flow Field for Consistent Image Restoration
• UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark
• WeatherCycle: Unpaired Multi-Weather Restoration via Color Space Decoupled Cycle Learning
• LucidFlux: Caption-Free Universal Image Restoration via a Large-Scale Diffusion Transformer
• BIR-Adapter: A Low-Complexity Diffusion Model Adapter for Blind Image Restoration
• LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter

Related

• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration
• A Unified Conditional Framework for Diffusion-based Image Restoration
• Diffusion Restoration Adapter for Real-World Image Restoration
• Exploiting Diffusion Prior for Task-driven Image Restoration
• EAM: Enhancing Anything with Diffusion Transformers for Blind Super-Resolution