Manifold Preserving Guided Diffusion

2023/11/28 by Yutong He, He, Yutong, Naoki Murata +19 · 46 citations
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2311.16424

openalex publication_date 2023/11/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Cited by

• FlowLPS: Langevin-Proximal Sampling for Flow-based Inverse Problem Solvers
• SURE Guided Posterior Sampling: Trajectory Correction for Diffusion-Based Inverse Problems
• Unified Control for Inference-Time Guidance of Denoising Diffusion Models
• Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation
• SONIC: Spectral Optimization of Noise for Inpainting with Consistency
• Enhanced Privacy Leakage from Noise-Perturbed Gradients via Gradient-Guided Conditional Diffusion Models
• Latent Refinement via Flow Matching for Training-free Linear Inverse Problem Solving
• Surflo: Consistent 3D Surface Flow Model with Global State
• Noise is All You Need: Solving Linear Inverse Problems by Noise Combination Sampling with Diffusion Models
• GOOD: Training-Free Guided Diffusion Sampling for Out-of-Distribution Detection
• Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction
• RNAGenScape: Property-guided Optimization and Interpolation of mRNA Sequences with Manifold Langevin Dynamics
• Coupled Data and Measurement Space Dynamics for Enhanced Diffusion Posterior Sampling
• Local MAP Sampling for Diffusion Models
• TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling
• Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL
• Training-Free Reward-Guided Image Editing via Trajectory Optimal Control
• GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
• DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models
• Diffusion models for inverse problems
• TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models
• Product of Experts for Visual Generation
• Stein Diffusion Guidance: Training-Free Posterior Correction for Sampling Beyond High-Density Regions
• Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models
• FRIDU: Functional Map Refinement with Guided Image Diffusion
• Exploiting the Exact Denoising Posterior Score in Training-Free Guidance of Diffusion Models
• Constraint-Guided Prediction Refinement via Deterministic Diffusion Trajectories
• Measurement-Aligned Sampling for Inverse Problem
• Multi-Step Guided Diffusion for Image Restoration on Edge Devices: Toward Lightweight Perception in Embodied AI
• Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models
• SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score
• Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach
• Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning
• Latent Guidance in Diffusion Models for Perceptual Evaluations
• Inference-time Scaling of Diffusion Models through Classical Search
• EquiReg: Equivariance Regularized Diffusion for Inverse Problems
• Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
• ExpertGen: Training-Free Expert Guidance for Controllable Text-to-Face Generation
• AudioMorphix: Training-free audio editing with diffusion probabilistic models
• LatentINDIGO: An INN-Guided Latent Diffusion Algorithm for Image Restoration
• Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variation
• Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction
• Towards SFW sampling for diffusion models via external conditioning
• Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps
• Image Interpolation with Score-based Riemannian Metrics of Diffusion Models
• Adaptive Diffusion Models for Sparse-View Motion-Corrected Head Cone-beam CT

Related

• Conditional Image Generation with Pretrained Generative Model
• Elucidating The Design Space of Classifier-Guided Diffusion Generation
• CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
• Towards Practical Plug-and-Play Diffusion Models
• Decoupling Training-Free Guided Diffusion by ADMM