Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment

2025/08/27 by Kim, Namu, Kweon, Wonbin, Kim, Minsoo +1
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.21090

Abstract

We observe that zero-shot appearance transfer with large-scale image generation models faces a significant challenge: Attention Leakage. This challenge arises when the semantic mapping between two images is captured by the Query-Key alignment. To tackle this issue, we introduce Q-Align, utilizing Query-Query alignment to mitigate attention leakage and improve the semantic alignment in zero-shot appearance transfer. Q-Align incorporates three core contributions: (1) Query-Query alignment, facilitating the sophisticated spatial semantic mapping between two images; (2) Key-Value rearrangement, enhancing feature correspondence through realignment; and (3) Attention refinement using rearranged keys and values to maintain semantic consistency. We validate the effectiveness of Q-Align through extensive experiments and analysis, and Q-Align outperforms state-of-the-art methods in appearance fidelity while maintaining competitive structure preservation.

Citations

• GPT-4o System Card
• FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing
• Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing
• ZONE: Zero-Shot Instruction-Guided Local Editing
• Cross-Image Attention for Zero-Shot Appearance Transfer
• Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
• DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
• Emergent Correspondence from Image Diffusion
• Segment Anything in High Quality
• A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
• Unsupervised Semantic Correspondence Using Stable Diffusion
• Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
• MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• ASIC: Aligning Sparse in-the-wild Image Collections
• Zero-shot Image-to-Image Translation
• Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
• DiffEdit: Diffusion-based semantic image editing with mask guidance
• Diffusion-based Image Translation using Disentangled Style and Content Representation
• Prompt-to-Prompt Image Editing with Cross Attention Control
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences
• CATs++: Boosting Cost Aggregation with Convolutions and Transformers
• Splicing ViT Features for Semantic Appearance Transfer
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Alignment Attention by Matching Key and Query Distributions
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
• StarGAN v2: Diverse Image Synthesis for Multiple Domains
• Attention Is All You Need
• A Neural Algorithm of Artistic Style
• Very Deep Convolutional Networks for Large-Scale Image Recognition

Related

• Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models
• Cross-Image Attention for Zero-Shot Appearance Transfer
• Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions
• SETR: A Two-Stage Semantic-Enhanced Framework for Zero-Shot Composed Image Retrieval
• SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer