SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

2025/02/25 by Jintao Zhang, Chendong Xiang, Zhang, Jintao +11 · 53 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Performance (cs.PF) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.18137

openalex publication_date 2025/02/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The code is available at https://github.com/thu-ml/SpargeAttn.

Cited by

• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
• CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
• Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
• Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10×
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
• Block Sparse Flash Attention
• Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence
• PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
• AVGGT: Rethinking Global Attention for Accelerating VGGT
• db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism
• Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration
• Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulation
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
• StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation
• SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
• SpecAttn: Speculating Sparse Attention
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
• Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
• Sparser Block-Sparse Attention via Token Permutation
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
• Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attention
• EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
• MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
• SkipSR: Faster Super Resolution with Token Skipping
• LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
• Towards Sampling Data Structures for Tensor Products in Turnstile Streams
• HilbertA: Hilbert Attention for Image Generation with Diffusion Models
• FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers
• DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencoder
• Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• RAPID³: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer
• FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
• DiCache: Let Diffusion Model Determine Its Own Cache
• Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers
• Dynamic Sparse Attention on Mobile SoCs
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• Mixture of Contexts for Long Video Generation
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
• BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation
• A Lightweight Learned Cardinality Estimation Model
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
• Efficient Attention Mechanisms for Large Language Models: A Survey

Related

• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
• Attention Condensation via Sparsity Induced Regularized Training
• SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention