Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

2024/03/26 by Donghoon Ahn, Ahn, Donghoon, Hyoungwon Cho +15 · 25 citations
Mathematics · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Survey Sampling and Estimation Techniques

paper · pdf · doi:10.48550/arxiv.2403.17377

openalex publication_date 2024/03/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms' ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.

Cited by

• EMAG: Self-Rectifying Diffusion Sampling with Exponential Moving Average Guidance
• DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoders
• CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models
• Guiding Visual Autoregressive Models through Spectrum Weakening
• Saddle-Free Guidance: Improved On-Manifold Sampling without Labels or Additional Training
• DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
• Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
• ACG: Action Coherence Guidance for Flow-based VLA models
• Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models
• CADE 2.5 - ZeResFDG: Frequency-Decoupled, Rescaled and Zero-Projected Guidance for SD/SDXL Latent Diffusion Models
• Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
• PickStyle: Video-to-Video Style Transfer with Context-Style Adapters
• MATRIX: Mask Track Alignment for Interaction-aware Video Generation
• TAG:Tangential Amplifying Guidance for Hallucination-Resistant Diffusion Sampling
• SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model
• FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation
• CrimEdit: Controllable Editing for Counterfactual Object Removal, Insertion, and Movement
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
• Noise-Level Diffusion Guidance: Well Begun is Half Done
• Composable Score-based Graph Diffusion Model for Multi-Conditional Molecular Generation
• TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement
• Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models
• SPG: Style-Prompting Guidance for Style-Specific Content Creation
• Increasing the Utility of Synthetic Images through Chamfer Guidance
• Steering Guidance for Personalized Text-to-Image Diffusion Models

Related

• Improving Sample Quality of Diffusion Models Using Self-Attention Guidance
• Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance
• EMAG: Self-Rectifying Diffusion Sampling with Exponential Moving Average Guidance
• Diffusion Sampling Path Tells More: An Efficient Plug-and-Play Strategy for Sample Filtering
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models