LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

2026/08/03 by Wen Zan, Jiaqi Zhang, Jianchao Tan +11
Computer Science · #cs.AI #cs.CL #cs.DC #cs.LG

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive O(L²) scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.

Citations

• GLM-5: from Vibe Coding to Agentic Engineering
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
• Towards Robust Mathematical Reasoning
• VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
• LongCat-Flash Technical Report
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
• BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
• TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
• HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
• RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Efficient Streaming Language Models with Attention Sinks
• CMMLU: Measuring massive multitask language understanding in Chinese
• Enabling Large Language Models to Generate Text with Citations
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities
• ASQA: Factoid Questions Meet Long-Form Answers
• Measuring Mathematical Problem Solving With the MATH Dataset
• Measuring Massive Multitask Language Understanding
• Big Bird: Transformers for Longer Sequences
• Longformer: The Long-Document Transformer
• Reformer: The Efficient Transformer
• Fast Transformer Decoding: One Write-Head is All You Need
• Generating Long Sequences with Sparse Transformers
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• The NarrativeQA Reading Comprehension Challenge
• Attention Is All You Need
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs