Manifold Preserving Guided Diffusion

2023/11/28 by He, Yutong, Murata, Naoki, Lai, Chieh-Hsin +8 · 21 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2311.16424

Abstract

Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Cited by

• FlowLPS: Langevin-Proximal Sampling for Flow-based Inverse Problem Solvers
• SURE Guided Posterior Sampling: Trajectory Correction for Diffusion-Based Inverse Problems
• Unified Control for Inference-Time Guidance of Denoising Diffusion Models
• Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation
• SONIC: Spectral Optimization of Noise for Inpainting with Consistency
• Enhanced Privacy Leakage from Noise-Perturbed Gradients via Gradient-Guided Conditional Diffusion Models
• Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solving
• Surflo: Consistent 3D Surface Flow Model with Global State
• Noise is All You Need: Solving Linear Inverse Problems by Noise Combination Sampling with Diffusion Models
• GOOD: Training-Free Guided Diffusion Sampling for Out-of-Distribution Detection
• Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
• RNAGenScape: Property-guided Optimization and Interpolation of mRNA Sequences with Manifold Langevin Dynamics
• Coupled Data and Measurement Space Dynamics for Enhanced Diffusion Posterior Sampling
• Local MAP Sampling for Diffusion Models
• TAG:Tangential Amplifying Guidance for Hallucination-Resistant Diffusion Sampling
• Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL
• Training-Free Reward-Guided Image Editing via Trajectory Optimal Control
• GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
• DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models
• Diffusion models for inverse problems
• TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models

Related

• Conditional Image Generation with Pretrained Generative Model
• Elucidating The Design Space of Classifier-Guided Diffusion Generation
• CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
• Towards Practical Plug-and-Play Diffusion Models
• Decoupling Training-Free Guided Diffusion by ADMM