VSA: Faster Video Diffusion with Trainable Sparse Attention

2025/05/19 by Yongqi Chen, Zhang, Peiyuan, Chen, Yongqi +11 · 32 citations
Computer Science · Medicine · #Advanced Neuroimaging Techniques and Applications #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2505.13389

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. In VSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computes token-level attention only inside those tiles subjecting to block computing layout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85% of FlashAttention3 MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA reaches a Pareto point that cuts training FLOPS by 2.53× with no drop in diffusion loss. Retrofitting the open-source Wan-2.1 model speeds up attention time by 6× and lowers end-to-end generation time from 31s to 18s with comparable quality. These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models. Code will be available at https://github.com/hao-ai-lab/FastVideo.

Citations

• Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light
• Wan: Open and Advanced Large-Scale Video Generative Models
• XAttention: Block Sparse Attention with Antidiagonal Scoring
• Training-free and Adaptive Sparse Attention for Efficient Long Video Generation
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
• Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
• Goku: Flow Based Video Generative Foundation Models
• Fast Video Generation with Sliding Tile Attention
• Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
• Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models
• Open-Sora: Democratizing Efficient Video Production for All
• Qwen2.5 Technical Report
• Flex Attention: A Programming Model for Generating Optimized Attention Kernels
• HunyuanVideo: A Systematic Framework For Large Video Generative Models
• Open-Sora Plan: Open-Source Large Video Generation Model
• ThunderKittens: Simple, Fast, and Adorable AI Kernels
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• The Llama 3 Herd of Models
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
• Improved Distribution Matching Distillation for Fast Image Synthesis
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Latte: Latent Diffusion Transformer for Video Generation
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• Efficient Streaming Language Models with Attention Sinks
• Effective Long-Context Scaling of Foundation Models
• LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
• BiFormer: Vision Transformer with Bi-Level Routing Attention
• Scalable Diffusion Models with Transformers
• Flow Matching for Generative Modeling
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training Compute-Optimal Large Language Models
• LongT5: Efficient Text-To-Text Transformer for Long Sequences
• Big Bird: Transformers for Longer Sequences
• Longformer: The Long-Document Transformer
• Efficient Content-Based Sparse Attention with Routing Transformers
• Scaling Laws for Neural Language Models
• Generating Long Sequences with Sparse Transformers
• Attention Is All You Need
• SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

Cited by

• MobileWan: Closing the Quality Gap for Mobile Video Diffusion
• Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
• End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
• Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10×
• USV: Unified Sparsification for Accelerating Video Diffusion Models
• PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
• Block Cascading: Training Free Acceleration of Block-Causal Video Models
• LongCat-Video Technical Report
• A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation
• MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
• NOSA: Native and Offloadable Sparse Attention
• FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution
• VideoNSA: Native Sparse Attention Scales Video Understanding
• LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
• OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot
• Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
• Training Agents Inside of Scalable World Models
• SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
• Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• Mixture of Contexts for Long Video Generation
• BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation
• Critique of World Model
• PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
• Dual-Expert Consistency Model for Efficient and High-Quality Video Generation
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
• Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention
• SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
• Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation

Related

• Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
• Fast Video Generation with Sliding Tile Attention