Bidirectional Sparse Attention for Faster Video Diffusion Training

2025/09/01 by Chenlu Zhan, Zhan, Chenlu, Wen J. Li +9 · 3 citations
Computer Science · Medicine · #Advanced Neuroimaging Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image and Video Quality Assessment

paper · pdf · doi:10.48550/arxiv.2509.01085

openalex publication_date 2025/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Video diffusion Transformer (DiT) models excel in generative quality but hit major computational bottlenecks when producing high-resolution, long-duration videos. The quadratic complexity of full attention leads to prohibitively high training and inference costs. Full attention inefficiency stems from two key challenges: excessive computation due to the inherent sparsity of Queries and Key-Value pairs, and redundant computation as fixed sparse patterns fail to leverage DiT's dynamic attention. To overcome this limitation, we propose a Bidirectional Sparse Attention (BSA) framework for faster video DiT training, the first to dynamically sparsify both Queries and Key-Value pairs within 3D full attention, thereby substantially improving training and inference efficiency. BSA addresses these issues through two key components. Query sparsity is optimized by selecting the most informative query tokens via semantic similarity and with a dynamic spatial-time training strategy, while KV sparsity is achieved by computing a statistical dynamic threshold to retain only the most salient KV blocks for computation. Extensive experiments demonstrate that BSA significantly accelerates DiT training across long sequences, reducing FLOPs by up to 20x and achieving 17.79x faster attention training, while preserving or even surpassing the generative quality of full attention.

Citations

• Mixture of Contexts for Long Video Generation
• VMoBA: Mixture-of-Block Attention for Video Diffusion Models
• Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
• SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
• VORTA: Efficient Video Diffusion via Routing Sparse Attention
• Training-Free Efficient Video Generation via Dynamic Token Carving
• Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
• Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light
• MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention
• Wan: Open and Advanced Large-Scale Video Generative Models
• FastVID: Dynamic Density Pruning for Fast Video Large Language Models
• Training-free and Adaptive Sparse Attention for Efficient Long Video Generation
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
• SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference
• LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
• DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
• Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
• FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generation
• MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation
• Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
• Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
• Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models
• MagicMirror: ID-Preserved Video Generation in Video Diffusion Transformers
• CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
• HunyuanVideo: A Systematic Framework For Large Video Generative Models
• SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
• Movie Gen: A Cast of Media Foundation Models
• Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
• DiTFastAttn: Attention Compression for Diffusion Transformer Models
• Latte: Latent Diffusion Transformer for Video Generation
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• Efficient Streaming Language Models with Attention Sinks
• LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• Scalable Diffusion Models with Transformers
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• LongT5: Efficient Text-To-Text Transformer for Long Sequences
• Longformer: The Long-Document Transformer
• Attention Is All You Need
• LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

Cited by

• DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoders
• USV: Unified Sparsification for Accelerating Video Diffusion Models
• NOSA: Native and Offloadable Sparse Attention

Related

• DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
• Training-free and Adaptive Sparse Attention for Efficient Long Video Generation
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
• DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance