Efficient Streaming Language Models with Attention Sinks

2023/09/29 by Guangxuan Xiao, Yuandong Tian, Xiao, Guangxuan +7 · 5 voices · 684 citations
Computer Science · #Algorithm #Artificial intelligence #Cache #Computer network #Computer science #Decoding methods #Language model #Natural Language Processing Techniques #Parallel computing #Security token #Sequence (biology) #Sink (geography) #Sliding window protocol #Speech Recognition and Synthesis #Speedup #Topic Modeling #Window (computing) #World Wide Web

paper · pdf · doi:10.48550/arxiv.2309.17453

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/09/29 · openalex created_date 2023/10/03 · openalex updated_date 2026/08/05

Abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Cited by

• DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding
• HiKV: Hierarchical Importance-Aware KV Cache With Hardware Acceleration for LLM Decoding
• PReM: Learning What to Preserve and When to Refresh for Context Compression
• HijackKV: New Threat in Position-Independent KV Cache Reuse
• Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference
• Self Gradient Forcing: Native Long Video Extrapolation
• Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
• Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space
• CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
• Test-Time Registers as Global Priors for Tokenized Image Generation
• Vidu S1: A Real-Time Interactive Video Generation Model
• An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
• Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
• Surprise Forcing: What to Remember, When to Skip in Long Video Generation
• The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
• When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
• Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
• Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs
• A Better Start for Language Models: Domain-Conditional Position Offsets
• Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
• DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction
• Hierarchical Denoising For Multi-Step Visual Reasoning
• VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
• InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
• Reflex: Real-Time VLA Control through Streaming Inference
• Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
• Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
• Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
• Self-Compacting Language Model Agents
• High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
• Still: Amortized KV Cache Compaction in a Single Forward Pass
• Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
• Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
• The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
• LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
• DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
• Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
• SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
• Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention
• Effective Distillation to Hybrid xLSTM Architectures
• Mitigating Conversational Inertia in Multi-Turn Agents
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Kimi Linear: An Expressive, Efficient Attention Architecture
• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models
• Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
• ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference
• StreamingVLM: Real-Time Understanding for Infinite Video Streams
• Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
• ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
• InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
• On the Existence and Behavior of Secondary Attention Sinks
• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• Attention Residuals
• Express Language Modeling
• StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
• Anchor Attention, Small Cache: Code Generation with Large Language Models
• Disentangling Semantic Attention from Structural Bias in the Attention Manifold
• TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
• KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems
• LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
• Memory for Large Language Models
• Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
• Tokengeist: Multi-Turn Attribution Tracing in Agentic Conversations
• MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
• AgentWatcher: A Rule-based Prompt Injection Monitor
• Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
• Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
• When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
• FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation
• Accelerate Speculative Decoding with Sparse Computation in Verification
• TimeBill: Time-Budgeted Inference for Large Language Models
• AstraNav-Memory: Contexts Compression for Long Memory
• HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
• ActionFlow: A Pipelined Action Acceleration for Vision Language Models on Edge
• MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding
• Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
• CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
• WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
• PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
• A Unified Sparse Attention via Multi-Granularity Compression
• Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processing
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Endless World: Real-Time 3D-Aware Long Video Generation
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
• Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
• JoyStreamer-Flash: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
• Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery: Sublinear Memory Growth for Efficient LLM Inference
• Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
• CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
• ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
• SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing
• InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
• Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networks
• SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
• Block Sparse Flash Attention
• Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
• Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models
• Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression
• Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
• Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts
• KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing
• Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
• PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
• KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
• Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models
• SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs
• SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• HTTM: Head-wise Temporal Token Merging for Faster VGGT
• Block Cascading: Training Free Acceleration of Block-Causal Video Models
• SelfMOTR: Revisiting MOTR with Self-Generating Detection Priors
• Low-Resolution Editing is All You Need for High-Resolution Editing
• RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generation
• Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
• Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
• Test-Time Temporal Sampling for Efficient MLLM Video Understanding
• Steering in the Shadows: Causal Amplification for Activation Space Attacks in Large Language Models
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
• Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
• Weight-sparse transformers have interpretable circuits
• OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs
• Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
• DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
• Optimizing Mixture of Block Attention
• A³: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
• DoPE: Denoising Rotary Position Embedding
• Sentence-Anchored Gist Compression for Long-Context LLMs
• MotionStream: Real-Time Video Generation with Interactive Motion Controls
• Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
• Order-Level Attention Similarity Across Language Models: A Latent Commonality
• SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
• Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks
• KV Cache Transform Coding for Compact Storage in LLM Inference
• FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
• Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• SpecAttn: Speculating Sparse Attention
• Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
• STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
• Polybasic Speculative Decoding Through a Theoretical Perspective
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
• AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
• Efficient Vocal Source Separation Through Windowed Sink Attention
• Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
• PureKV: Plug-and-Play KV Cache Optimization with Spatial-Temporal Sparse Attention for Vision-Language Large Models
• FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference
• Serve Programs, Not Prompts
• Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
• Cache Merging as a Convergent Replicated State for Multi-Agent Latent Reasoning
• LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
• NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
• RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
• Mixture-of-Depths Attention
• Attention-Aligned Reasoning for Large Language Models
• Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• Simplified Sparse Attention via Gist Tokens
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• RGMem: Renormalization Group-inspired Memory Evolution for Language Agents
• Long-Context Modeling with Dynamic Hierarchical Sparse Attention for On-Device LLMs
• SALS: Sparse Attention in Latent Space for KV cache Compression
• Pie: A Programmable Serving System for Emerging LLM Applications
• Knocking-Heads Attention
• Can Language Models Compose Skills In-Context?
• Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
• Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs
• Efficient Low Rank Attention for Long-Context Inference in Large Language Models
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
• Chronos-2: From Univariate to Universal Forecasting
• Compressing Many-Shots in In-Context Learning
• Sparser Block-Sparse Attention via Token Permutation
• Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
• Attention Sinks in Diffusion Language Models
• Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
• No Mean Feat: Simple, Strong Baselines for Context Compression
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
• CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation
• SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation
• Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
• VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
• Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
• Learning to play: A Multimodal Agent for 3D Game-Play
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
• Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
• NOSA: Native and Offloadable Sparse Attention
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
• Taming the Fragility of KV Cache Eviction in LLM Inference
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
• APCE: Adaptive Progressive Context Expansion for Long Context Processing
• DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
• Dimension-Free Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
• VideoNSA: Native Sparse Attention Scales Video Understanding
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
• Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
• XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
• ShishuLM: Lightweight Language Model with Hybrid Decoder-MLP Architecture and Paired Weight Sharing
• DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• CacheClip: Accelerating RAG with Effective KV Cache Reuse
• MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
• AURA: Action-Gated Memory for Robot Policies at Constant VRAM
• MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
• Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
• DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
• OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• Prepared mind, fast response: A temporal decoupling framework for adaptive knowledge orchestration in open-domain dialogue
• Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
• Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models
• Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
• Revisiting Long-context Modeling from Context Denoising Perspective
• vAttention: Verified Sparse Attention
• Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMs
• Staircase Streaming for Low-Latency Multi-Agent Inference
• SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• Scaling Sequence-to-Sequence Generative Neural Rendering
• PatternKV: Flattening KV Representation Expands Quantization Headroom
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models
• Towards Sampling Data Structures for Tensor Products in Turnstile Streams
• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
• MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
• The Pitfalls of KV Cache Compression
• FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
• ProxyAttn: Guided Sparse Attention via Representative Heads
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Models
• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models
• Efficient Turing Machine Simulation with Transformers
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• RefAM: Attention Magnets for Zero-Shot Referral Segmentation
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• Learning the Neighborhood: Contrast-Free Multimodal Self-Supervised Molecular Graph Pretraining
• Investigating Faithfulness in Large Audio Language Models
• Review of Hallucination Understanding in Large Language and Vision Models
• We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule
• Efficient On-Device Agents via Adaptive Context Management
• Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
• Mamba Modulation: On the Length Generalization of Mamba
• Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series
• Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attention
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
• SemPIC: Learning Semantic Position-Independent KV Caches
• Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
• Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
• How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
• Defusing the Trigger: Tail-Risk-Informed Attention Rebalancing for LLM Backdoor Mitigation
• SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
• ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
• EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
• Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• Cross-Attention is Half Explanation in Speech-to-Text Models
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• BEFT: Bias-Efficient Fine-Tuning of Language Models
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens
• LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
• EvolKV: Evolutionary KV Cache Compression for LLM Inference
• SpikingBrain: Spiking Brain-inspired Large Models
• PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
• Adaptive KV-Cache Compression without Manually Setting Budget
• REFRAG: Rethinking RAG based Decoding
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
• GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
• Joint Enhancement of Relational Reasoning for Long-Context LLMs
• Mixture of Contexts for Long Video Generation
• MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning
• AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
• Robustness is Important: Limitations of LLMs for Data Fitting
• Autoregressive Universal Video Segmentation Model
• Integral Transformer: Denoising Attention, Not Too Much Not Too Little
• SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
• Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation
• H2EAL: Hybrid-Bonding Architecture with Hybrid Sparse Attention for Efficient Long-Context LLM Inference
• DPad: Efficient Diffusion Language Models with Suffix Dropout
• CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
• ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
• XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization
• Retrospective Sparse Attention for Efficient Long-Context Generation
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• OverFill: Two-Stage Models for Efficient Language Model Decoding
• DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attention
• Many-Turn Jailbreaking
• gpt-oss-120b & gpt-oss-20b Model Card
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
• Cognitive Workspace: Active Memory Management for LLMs -- An Empirical Study of Functional Infinite Context
• Attention Basin: Why Contextual Position Matters in Large Language Models
• Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
• Sparse Attention across Multiple-context KV Cache
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
• AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
• Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
• LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training
• Free-MoRef: Instantly Multiplexing Context Perception Capabilities of Video-MLLMs within Single Inference
• CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
• LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
• What are you sinking? A geometric approach on attention sink
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
• Polymorphic Combinatorial Frameworks (PCF): Guiding the Design of Mathematically-Grounded, Adaptive AI Agents
• PiKV: KV Cache Management System for Mixture of Experts
• EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devices
• DiffLoRA: Differential Low-Rank Adapters for Large Language Models
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
• TriangleMix: Accelerating Prefilling via Decoding-time Contribution Sparsity
• MemShare: Memory Efficient Inference for Large Reasoning Models through KV Cache Reuse
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
• CaliDrop: KV Cache Compression with Calibration
• DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
• Efficient Attention Mechanisms for Large Language Models: A Survey
• Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
• FOCUS: Fused Observation of Channels for Unveiling Spectra
• Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
• SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
• Docopilot: Improving Multimodal Models for Document-Level Understanding
• Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
• LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues
• Synergy: End-to-end Concept Model
• On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
• IAM: Efficient Inference through Attention Mapping between Different-scale LLMs
• LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
• Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition
• Draft-based Approximate Inference for LLMs
• Efficient Molecular Conformer Generation with SO(3)-Averaged Flow Matching and Reflow
• SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
• Lizard: An Efficient Linearization Framework for Large Language Models
• BlindSight: Harnessing Sparsity for Efficient Vision-Language Models
• Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
• Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
• SAS: Simulated Attention Score
• Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought
• KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
• Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
• SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
• Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
• SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design
• PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
• Selective KV Cache Protection for Noise-Resilient LLM Inference on Analog Compute-In-Memory Systems
• WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
• OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
• DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation
• MemOS: A Memory OS for AI System
• HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
• Understanding and Improving Length Generalization in Recurrent Models
• Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
• MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
• High-Layer Attention Pruning with Rescaling
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism
• Stem: Rethinking Causal Information Flow in Sparse Attention
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
• VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator
• Linearly Decoding Refused Knowledge in Aligned Language Models
• Pipelined Decoder for Efficient Context-Aware Text Generation
• Token Activation Map to Visually Explain Multimodal LLMs
• Lost at the Beginning of Reasoning
• A Survey of LLM Inference Systems
• Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation
• scMamba: A Scalable Foundation Model for Single-Cell Multi-Omics Integration Beyond Highly Variable Feature Selection
• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
• Vision Transformers Don't Need Trained Registers
• MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts
• Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
• AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
• MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection
• Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
• Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
• Steering Conceptual Bias via Transformer Latent-Subspace Activation
• MiniCPM4: Ultra-Efficient LLMs on End Devices
• PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries
• Parallel Continuous Chain-of-Thought with Jacobi Iteration
• Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models
• Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generation
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
• LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
• PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
• Long-Context Generalization with Sparse Attention
• Over-squashing in Spatiotemporal Graph Neural Networks
• eLLM: Elastic Memory Management Framework for Efficient LLM Serving
• Think Clearly: Improving Reasoning via Redundant Token Pruning
• Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systems
• PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
• Multipole Attention for Efficient Long Context Reasoning
• LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
• Two Heads Are Better than One: Simulating Large Transformers with Small Ones
• Lag-Relative Sparse Attention In Long Context Training
• CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
• Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
• United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory
• DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
• MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
• CoMemo: LVLMs Need Image Context with Image Memory
• Cartridges: Lightweight and general-purpose long context representations via self-study
• Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
• SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
• Sample Complexity and Representation Ability of Test-time Scaling Paradigms
• Inference-Time Hyper-Scaling with KV Cache Compression
• Kinetics: Rethinking Test-Time Scaling Laws
• GEM: Empowering LLM for both Embedding Generation and Language Understanding
• SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
• LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders
• AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models
• Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
• HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference
• METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding
• Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
• KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
• Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
• Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
• Scaling Textual Gradients via Sampling-Based Momentum
• How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
• Vid2Coach: Transforming How-To Videos into Task Assistants
• EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
• A Simple Linear Patch Revives Layer-Pruned Large Language Models
• Mamba Knockout for Unraveling Factual Information Flow
• Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
• SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
• The Surprising Soupability of Documents in State Space Models
• TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
• AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
• RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference
• Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
• FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference
• Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
• Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?
• SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
• SageAttention2++: A More Efficient Implementation of SageAttention2
• Exploring the Hidden Capacity of LLMs for One-Step Text Generation
• Vision Transformers with Self-Distilled Registers
• Hardware-Efficient Attention for Fast Decoding
• R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
• HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
• AdaTP: Attention-Debiased Token Pruning for Video Large Language Models
• Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
• FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
• Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing
• Efficient Multi-modal Long Context Learning for Training-free Adaptation
• TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
• HAMburger: Accelerating LLM Inference via Token Smashing
• FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
• 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression
• Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs
• Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
• Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
• PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
• Rethinking Causal Mask Attention for Vision-Language Inference
• Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
• Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
• SELF: Self-Extend the Context Length With Logistic Growth Function
• Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoning
• The First Impression Problem: Internal Bias Triggers Overthinking in Reasoning Models
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
• MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention
• NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
• QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design
• LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding
• DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
• The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
• Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
• LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
• LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
• RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
• QA-prompting: Improving Summarization with Large Language Models using Question-Answering
• Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
• Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
• Mechanistic Fine-tuning for In-context Learning
• Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
• FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference
• Fractured Chain-of-Thought Reasoning
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
• Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
• ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning
• GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment
• KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
• FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge
• Context Compaction Theory
• DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
• Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning
• dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction
• Accurate KV Cache Quantization with Outlier Tokens Tracing
• ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
• Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents
• SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
• SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
• S⁴R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
• SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
• What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs
• Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
• Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
• Geometric Analysis of Token Selection in Multi-Head Attention
• Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data
• The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)
• Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence
• TurboESM: Ultra-Efficient 3-Bit KV Cache Quantization for Protein Language Models with Orthogonal Rotation and QJL Correction
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
• Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
• Taming Outlier Tokens in Diffusion Transformers
• Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
• Fast and Low-Cost Genomic Foundation Models via Outlier Removal
• FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension
• ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
• Sphere Retraction Normalizations
• ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
• FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion
• Epiphany-Aware KV Cache Eviction Without the Attention Matrix
• RoPE-Aware Bit Allocation for KV-Cache Quantization
• AtomMem: Building Simple and Effective Memory System for LLM Agents via Atomic Facts
• MiniMax Sparse Attention
• On Subquadratic Architectures: From Applications to Principles
• From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
• The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
• Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
• RLHFless: Serverless Computing for Efficient RLHF
• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
• SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
• KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
• From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
• SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
• Self-GC: Self-Governing Context for Long-Horizon LLM Agents
• Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
• Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
• Gated Memory Policy: In-Context Memorization and Adaptation
• The Key to Going Linear: Analysis-Driven Transformer Linearization
• Learning Agent-Compatible Context Management for Long-Horizon Tasks
• Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
• Let ViT Speak: Generative Language-Image Pre-training
• LongStream: Long-Sequence Streaming Autoregressive Visual Geometry
• Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
• Formal Verification of TurboQuant: Machine-Checked Proofs and Gap Closures
• HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
• Neural Garbage Collection: Learning to Forget while Learning to Reason
• Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
• Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory
• Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
• FOCUS: DLLMs Know How to Tame Their Compute Bound
• You Need Better Attention Priors
• ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
• Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck
• System-Mediated Attention Imbalances Make Vision-Language Models Say Yes
• Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
• Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
• Attention Needs to Focus: A Unified Perspective on Attention Allocation
• TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
• SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
• HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
• AnchorKV: Anchor-Residual KV Cache Compression
• An Empirical Study on Prompt Compression for Large Language Models
• VoxStruct3D: Structure-Leading Flow Matching for Voxel-Space 3D MRI Synthesis
• Training-Free Hashing-Based Attention via Binary Principal Components
• Identifying and Evaluating Inactive Heads in Pretrained LLMs
• Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
• When More Becomes Less: Position-Dependent Repetition Effects in Language Models
• PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation
• Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection
• Saliency-driven Dynamic Token Pruning for Large Language Models
• The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Study
• LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
• MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention
• Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
• From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
• KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
• Efficient Pretraining Length Scaling
• Efficient Function Orchestration for Large Language Models
• CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
• Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs
• Runtime Observability for Heterogeneous Attention Memory
• Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
• QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
• Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference
• The Impossibility Triangle of Long-Context Modeling
• Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering
• Layer-wise Positional Bias in Short-Context Language Modeling
• HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
• KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
• AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference
• Analysis of Attention in Video Diffusion Transformers
• Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMs
• Long Context In-Context Compression by Getting to the Gist of Gisting
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
• Mimic In-Context Learning for Multimodal Tasks
• UniCAIM: A Unified CAM/CIM Architecture with Static-Dynamic KV Cache Pruning for Efficient Long-Context LLM Inference
• Adaptive Computation Pruning for the Forgetting Transformer
• EDIT: Enhancing Vision Transformers by Mitigating Attention Sink through an Encoder-Decoder Architecture
• Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
• SPIRe: Boosting LLM Inference Throughput with Speculative Decoding

Discussions

• StreamingLLM: Efficient streaming technique enable infinite sequence lengths [hn, 118 points, 12 comments]
• Efficient Streaming Language Models with Attention Sinks [hn, 5 points, 0 comments]
• Efficient Streaming Language Models with Attention Sinks [hn, 2 points, 0 comments]
• I finally read the attention sink paper [1] and the HF blog article [2]. Seems like another interesting data point that the models we usually interact with strongly attend to the first few tokens... [ [bsky, 0 points, 0 comments]
• This turns out to be a special case of the attention sink as implemented in the new GPT models, as explained in this paper: arxiv.org/pdf/2309.174... [bsky, 0 points, 1 comments]

Related

• Streaming Attention Approximation via Discrepancy Theory
• When Attention Sink Emerges in Language Models: An Empirical View
• Why do LLMs attend to the first token?
• Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration