Universal Guidance for Diffusion Models

2023/02/14 by Arpit Bansal, Bansal, Arpit, Hongmin Chu +11 · 50 citations
Computer Science · Medicine · #Generative Adversarial Networks and Image Synthesis #Advanced Mathematical Modeling in Engineering #Advanced Neuroimaging Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2302.07121

Abstract

Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available at https://github.com/arpitbansal297/Universal-Guided-Diffusion.

Cited by

• Human-Aligned Generative Perception: Bridging Psychophysics and Generative Models
• Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance
• Unified Control for Inference-Time Guidance of Denoising Diffusion Models
• Training-free Clothing Region of Interest Self-correction for Virtual Try-On
• 3DID: Direct 3D Inverse Design for Aerodynamics with Physics-Aware Optimization
• Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints
• CoDA: From Text-to-Image Diffusion Models to Training-Free Dataset Distillation
• 3D-Consistent Multi-View Editing by Correspondence Guidance
• Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation
• DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
• MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization
• SONIC: Spectral Optimization of Noise for Inpainting with Consistency
• Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion
• Show Me: Unifying Instructional Image and Video Generation with Diffusion Models
• InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization
• HardFlow: Hard-Constrained Sampling for Flow-Matching Models via Trajectory Optimization
• Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
• Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
• Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solving
• Scaling Multi-Agent Environment Co-Design with Diffusion Models
• LumaGuide: Distribution Shaping for Training-Free HDR Generation in Diffusion Models
• Surflo: Consistent 3D Surface Flow Model with Global State
• Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models
• Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
• GOOD: Training-Free Guided Diffusion Sampling for Out-of-Distribution Detection
• GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
• Coupled Diffusion Sampling for Training-Free Multi-View Image Editing
• Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
• T(R,O) Grasp: Efficient Graph Diffusion of Robot-Object Spatial Transformation for Cross-Embodiment Dexterous Grasping
• Discrete State Diffusion Models: A Sample Complexity Perspective
• Local MAP Sampling for Diffusion Models
• C3Editor: Achieving Controllable Consistency in 2D Model for 3D Editing
• Flexible Locomotion Learning with Diffusion Model Predictive Control
• Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL
• Diffusion Alignment as Variational Expectation-Maximization
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
• DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models
• PolyJuice Makes It Real: Black-Box, Universal Red Teaming for Synthetic Image Detectors
• DRAG: Data Reconstruction Attack using Guided Diffusion
• Feature Space Analysis by Guided Diffusion Model
• CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models
• TeRA: Rethinking Text-guided Realistic 3D Avatar Generation
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance
• Conformal Predictive Monitoring for Multi-Modal Scenarios
• Feynman-Kac-Flow: Inference Steering of Conditional Flow Matching to an Energy-Tilted Posterior
• SAGA: Learning Signal-Aligned Distributions for Improved Text-to-Image Generation
• Enhancing Diffusion-based Dataset Distillation via Adversary-Guided Curriculum Sampling
• YOLO-Count: Differentiable Object Counting for Text-to-Image Generation
• TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models

Related

• Diffusion Models without Classifier-free Guidance
• Understanding and Improving Training-free Loss-based Diffusion Guidance
• How Much To Guide: Revisiting Adaptive Guidance in Classifier-Free Guidance Text-to-Vision Diffusion Models
• Self-Guided Diffusion Models
• Segmentation-Free Guidance for Text-to-Image Diffusion Models