NOSA: Native and Offloadable Sparse Attention

2025/10/15 by Yuxiang Huang, Wang, Pengjie, Huang, Yuxiang +10 · 1 citation
Computer Science · Engineering · Medicine · #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics and Automated Systems #Seismology and Earthquake Studies

paper · pdf · doi:10.48550/arxiv.2510.13602

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Decoding throughput improvements from larger inference batches are limited by GPU memory, which is largely consumed by the key-value (KV) cache. Prior training-free KV cache offloading alleviates this by keeping redundant context on the CPU and fetching only a sparse subset for attention, but it often degrades long-generation quality due to training-inference mismatch on sparse patterns. Meanwhile, trainable sparse attention is incompatible with efficient offloading, as unconstrained KV accesses may force large CPU-to-GPU transfers and erase throughput gains. To this end, we propose NOSA, a trainable sparse attention mechanism natively designed for KV cache offloading. NOSA explicitly constrains the volume of CPU-GPU KV transfers, thereby achieving low communication overhead and high decoding throughput. We further build NOSI, a KV cache offloading inference system that fully unlocks NOSA's efficiency. Empirical results on 1,3,8B LLMs demonstrate that NOSA outperforms KV cache offloading baselines on general, long-input, and long-generation tasks, while boosting decoding throughput by up to 5.04x, 1.92x, and 1.83x over FullAttn, InfLLMv2, and ShadowKV, respectively. We release our code at https://github.com/thunlp/NOSA.

Citations

• ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
• PureKV: Plug-and-Play KV Cache Optimization with Spatial-Temporal Sparse Attention for Vision-Language Large Models
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• Sparse Attention across Multiple-context KV Cache
• Trainable Dynamic Mask Sparse Attention
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
• Long-Context Generalization with Sparse Attention
• MiniCPM4: Ultra-Efficient LLMs on End Devices
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• SageAttention2++: A More Efficient Implementation of SageAttention2
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
• Hierarchical Sparse Attention Framework for Computationally Efficient Classification of Biological Cells
• R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
• MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention
• Large Language Model Agent: A Survey on Methodology, Applications and Challenges
• XAttention: Block Sparse Attention with Antidiagonal Scoring
• LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
• Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
• MoBA: Mixture of Block Attention for Long-Context LLMs
• AdaSplash: Adaptive Sparse Flash Attention
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
• SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
• Squeezed Attention: Accelerating Long Context Length LLM Inference
• Anchor Attention, Small Cache: Code Generation with Large Language Models
• ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
• GPT-4o System Card
• MagicPIG: LSH Sampling for Efficient LLM Generation
• AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
• SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
• SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
• HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
• Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
• InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
• Post-Training Sparse Attention with Double Sparsity
• Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
• The Llama 3 Herd of Models
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
• QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
• Loki: Low-rank Keys for Efficient Sparse Attention
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
• SirLLM: Streaming Infinite Retentive LLM
• Layer-Condensed KV Cache for Efficient Inference of Large Language Models
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• SnapKV: LLM Knows What You are Looking for Before Generation
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• LLM Inference Unveiled: Survey and Roofline Model Insights
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
• KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• SGLang: Efficient Execution of Structured Language Model Programs
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Efficient Streaming Language Models with Attention Sinks
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• Let's Verify Step by Step
• Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark
• Tool Learning with Foundation Models
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Compressive Transformers for Long-Range Sequence Modelling
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Attention Is All You Need

Cited by

• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention

Related

• Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
• Post-Training Sparse Attention with Double Sparsity
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space