Efficient Streaming Language Models with Attention Sinks

2023/09/29 by Guangxuan Xiao, Yuandong Tian, Xiao, Guangxuan +7 · 5 voices · 388 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2309.17453

Abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Cited by

• DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding
• HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
• PReM: Learning What to Preserve and When to Refresh for Context Compression
• HijackKV: New Threat in Position-Independent KV Cache Reuse
• Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference
• Self Gradient Forcing: Native Long Video Extrapolation
• Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
• Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space
• CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
• Test-Time Registers as Global Priors for Tokenized Image Generation
• Vidu S1: A Real-Time Interactive Video Generation Model
• An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
• Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
• Surprise Forcing: What to Remember, When to Skip in Long Video Generation
• The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
• When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
• Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
• Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs
• A Better Start for Language Models: Domain-Conditional Position Offsets
• Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
• DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction
• Hierarchical Denoising For Multi-Step Visual Reasoning
• VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
• InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
• Reflex: Real-Time VLA Control through Streaming Inference
• Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
• Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
• Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
• Self-Compacting Language Model Agents
• High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
• Still: Amortized KV Cache Compaction in a Single Forward Pass
• Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
• Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
• The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
• LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
• DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
• Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
• SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
• Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention
• Effective Distillation to Hybrid xLSTM Architectures
• Mitigating Conversational Inertia in Multi-Turn Agents
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Kimi Linear: An Expressive, Efficient Attention Architecture
• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models
• Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
• ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference
• StreamingVLM: Real-Time Understanding for Infinite Video Streams
• Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
• ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
• InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
• On the Existence and Behavior of Secondary Attention Sinks
• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• Attention Residuals
• Express Language Modeling
• StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
• Anchor Attention, Small Cache: Code Generation with Large Language Models
• Disentangling Semantic Attention from Structural Bias in the Attention Manifold
• TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
• KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems
• LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
• Memory for Large Language Models
• Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
• Tokengeist: Multi-Turn Attribution Tracing in Agentic Conversations
• MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
• AgentWatcher: A Rule-based Prompt Injection Monitor
• Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
• Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
• When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
• FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation
• Accelerate Speculative Decoding with Sparse Computation in Verification
• TimeBill: Time-Budgeted Inference for Large Language Models
• AstraNav-Memory: Contexts Compression for Long Memory
• HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
• ActionFlow: A Pipelined Action Acceleration for Vision Language Models on Edge
• MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding
• Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
• CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
• WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
• PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
• A Unified Sparse Attention via Multi-Granularity Compression
• Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processing
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Endless World: Real-Time 3D-Aware Long Video Generation
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
• Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
• JoyAvatar: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
• Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery: Sublinear Memory Growth for Efficient LLM Inference
• Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
• CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
• ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
• SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing
• InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
• Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networks
• SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
• Block Sparse Flash Attention
• Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
• Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models
• Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression
• Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
• Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts
• KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing
• Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
• PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
• KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
• Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models
• SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs
• SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• HTTM: Head-wise Temporal Token Merging for Faster VGGT
• Block Cascading: Training Free Acceleration of Block-Causal Video Models
• SelfMOTR: Revisiting MOTR with Self-Generating Detection Priors
• Low-Resolution Editing is All You Need for High-Resolution Editing
• RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generation
• Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
• Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
• Test-Time Temporal Sampling for Efficient MLLM Video Understanding
• Steering in the Shadows: Causal Amplification for Activation Space Attacks in Large Language Models
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
• Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
• Weight-sparse transformers have interpretable circuits
• OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs
• Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
• DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
• Optimizing Mixture of Block Attention
• A³: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
• DoPE: Denoising Rotary Position Embedding
• Sentence-Anchored Gist Compression for Long-Context LLMs
• MotionStream: Real-Time Video Generation with Interactive Motion Controls
• Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
• Order-Level Attention Similarity Across Language Models: A Latent Commonality
• SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
• Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks
• KV Cache Transform Coding for Compact Storage in LLM Inference
• FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
• Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• SpecAttn: Speculating Sparse Attention
• Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
• STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
• Polybasic Speculative Decoding Through a Theoretical Perspective
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
• AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
• Efficient Vocal Source Separation Through Windowed Sink Attention
• Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
• PureKV: Plug-and-Play KV Cache Optimization with Spatial-Temporal Sparse Attention for Vision-Language Large Models
• FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference
• Serve Programs, Not Prompts
• Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
• Cache Merging as a Convergent Replicated State for Multi-Agent Latent Reasoning
• LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
• NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
• RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
• Mixture-of-Depths Attention
• Attention-Aligned Reasoning for Large Language Models
• Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• Simplified Sparse Attention via Gist Tokens
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• RGMem: Renormalization Group-inspired Memory Evolution for Language Agents
• Long-Context Modeling with Dynamic Hierarchical Sparse Attention for On-Device LLMs
• SALS: Sparse Attention in Latent Space for KV cache Compression
• Pie: A Programmable Serving System for Emerging LLM Applications
• Knocking-Heads Attention
• Can Language Models Compose Skills In-Context?
• Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
• Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs
• Efficient Low Rank Attention for Long-Context Inference in Large Language Models
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
• Chronos-2: From Univariate to Universal Forecasting
• Compressing Many-Shots in In-Context Learning
• Sparser Block-Sparse Attention via Token Permutation
• Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
• Attention Sinks in Diffusion Language Models
• Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
• Simple Context Compression: Mean-Pooling and Multi-Ratio Training
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
• CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation
• SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation
• Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
• VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
• Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
• Learning to play: A Multimodal Agent for 3D Game-Play
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
• Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
• NOSA: Native and Offloadable Sparse Attention
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
• Taming the Fragility of KV Cache Eviction in LLM Inference
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
• APCE: Adaptive Progressive Context Expansion for Long Context Processing
• DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
• Dimension-Free Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
• VideoNSA: Native Sparse Attention Scales Video Understanding
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
• Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
• XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
• ShishuLM: Lightweight Language Model with Hybrid Decoder-MLP Architecture and Paired Weight Sharing
• DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• CacheClip: Accelerating RAG with Effective KV Cache Reuse
• MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
• AURA: Action-Gated Memory for Robot Policies at Constant VRAM
• MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
• Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
• DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
• OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• Prepared mind, fast response: A temporal decoupling framework for adaptive knowledge orchestration in open-domain dialogue
• Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
• Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models
• Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
• Revisiting Long-context Modeling from Context Denoising Perspective
• vAttention: Verified Sparse Attention
• Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMs
• Staircase Streaming for Low-Latency Multi-Agent Inference
• SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• Scaling Sequence-to-Sequence Generative Neural Rendering
• PatternKV: Flattening KV Representation Expands Quantization Headroom
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models
• Towards Sampling Data Structures for Tensor Products in Turnstile Streams
• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
• MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
• The Pitfalls of KV Cache Compression
• FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
• ProxyAttn: Guided Sparse Attention via Representative Heads
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Models
• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models
• Efficient Turing Machine Simulation with Transformers
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• RefAM: Attention Magnets for Zero-Shot Referral Segmentation
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining
• Investigating Faithfulness in Large Audio Language Models
• Review of Hallucination Understanding in Large Language and Vision Models
• We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule
• Efficient On-Device Agents via Adaptive Context Management
• Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
• Mamba Modulation: On the Length Generalization of Mamba
• Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series
• Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attention
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
• SemPIC: Learning Semantic Position-Independent KV Caches
• Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
• Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
• How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
• Defusing the Trigger: Tail-Risk-Informed Attention Rebalancing for LLM Backdoor Mitigation
• SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
• ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
• EpiCache: Episodic KV Cache Management for Long Conversational Question Answering
• Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• Cross-Attention is Half Explanation in Speech-to-Text Models
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• BEFT: Bias-Efficient Fine-Tuning of Language Models
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens
• LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
• EvolKV: Evolutionary KV Cache Compression for LLM Inference
• SpikingBrain: Spiking Brain-inspired Large Models
• PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
• Adaptive KV-Cache Compression without Manually Setting Budget
• REFRAG: Rethinking RAG based Decoding
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
• GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
• Joint Enhancement of Relational Reasoning for Long-Context LLMs
• Mixture of Contexts for Long Video Generation
• MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning
• AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
• Robustness is Important: Limitations of LLMs for Data Fitting
• Autoregressive Universal Video Segmentation Model
• Integral Transformer: Denoising Attention, Not Too Much Not Too Little
• SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
• Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation
• H2EAL: Hybrid-Bonding Architecture with Hybrid Sparse Attention for Efficient Long-Context LLM Inference
• DPad: Efficient Diffusion Language Models with Suffix Dropout
• CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
• ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
• XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization
• Retrospective Sparse Attention for Efficient Long-Context Generation
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• OverFill: Two-Stage Models for Efficient Language Model Decoding
• DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attention
• Many-Turn Jailbreaking
• gpt-oss-120b & gpt-oss-20b Model Card
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
• Cognitive Workspace: Active Memory Management for LLMs -- An Empirical Study of Functional Infinite Context
• Attention Basin: Why Contextual Position Matters in Large Language Models
• Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
• Sparse Attention across Multiple-context KV Cache
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
• AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
• Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
• LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training
• Free-MoRef: Instantly Multiplexing Context Perception Capabilities of Video-MLLMs within Single Inference
• CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
• LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
• What are you sinking? A geometric approach on attention sink
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
• Polymorphic Combinatorial Frameworks (PCF): Guiding the Design of Mathematically-Grounded, Adaptive AI Agents
• PiKV: KV Cache Management System for Mixture of Experts
• EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devices
• DiffLoRA: Differential Low-Rank Adapters for Large Language Models
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
• TriangleMix: Accelerating Prefilling via Decoding-time Contribution Sparsity
• MemShare: Memory Efficient Inference for Large Reasoning Models through KV Cache Reuse
• A Survey of Token Compression for Efficient Multimodal Large Language Models

Discussions

• StreamingLLM: Efficient streaming technique enable infinite sequence lengths [hn, 118 points, 12 comments]
• Efficient Streaming Language Models with Attention Sinks [hn, 5 points, 0 comments]
• Efficient Streaming Language Models with Attention Sinks [hn, 2 points, 0 comments]
• I finally read the attention sink paper [1] and the HF blog article [2]. Seems like another interesting data point that the models we usually interact with strongly attend to the first few tokens... [ [bsky, 0 points, 0 comments]
• This turns out to be a special case of the attention sink as implemented in the new GPT models, as explained in this paper: arxiv.org/pdf/2309.174... [bsky, 0 points, 1 comments]

Related

• Streaming Attention Approximation via Discrepancy Theory
• When Attention Sink Emerges in Language Models: An Empirical View
• Why do LLMs attend to the first token?
• Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration