Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

2024/04/10 by Tsendsuren Munkhdalai, Manaal Faruqui, Munkhdalai, Tsendsuren +3 · 5 voices · 60 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computer science #Context (archaeology) #Electrical engineering #Engineering #History #Transformer

paper · pdf · doi:10.48550/arxiv.2404.07143

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/04/10 · openalex created_date 2024/04/12 · openalex updated_date 2026/07/28

Abstract

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

Cited by

• Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
• HiCI: Hierarchical Construction-Integration for Long-Context Attention
• Memory Caching: RNNs with Growing Memory
• Recursive Language Models
• Kimi Linear: An Expressive, Efficient Attention Architecture
• Log-Linear Attention
• Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
• Efficient Reasoning with Hidden Thinking
• GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
• AstraNav-Memory: Contexts Compression for Long Memory
• Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifolds
• Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments
• CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
• GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory
• Can We Go Beyond Visual Features? Neural Tissue Relation Modeling for Relational Graph Analysis in Non-Melanoma Skin Histology
• Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
• MossNet: Mixture of State-Space Experts is a Multi-Head Attention
• Simulating Subjects: The Promise and Peril of Artificial Intelligence Stand-Ins for Social Agents and Interactions
• Long-Context Modeling with Dynamic Hierarchical Sparse Attention for On-Device LLMs
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• Native Hybrid Attention for Efficient Sequence Modeling
• Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
• ResFormer: All-Time Reservoir Memory for Long Sequence Classification
• HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
• Context Recycling for Long-Horizon LLM Inference
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• Bounded PCTL Model Checking of Large Language Model Outputs
• Long-Context Speech Synthesis with Context-Aware Memory
• Cognitive Workspace: Active Memory Management for LLMs -- An Empirical Study of Functional Infinite Context
• EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devices
• Flora: Effortless Context Construction to Arbitrary Length and Scale
• Lizard: An Efficient Linearization Framework for Large Language Models
• White-Basilisk: A Hybrid Model for Code Vulnerability Detection
• MIRIX: Multi-Agent Memory System for LLM-Based Agents
• SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression
• A Survey on Latent Reasoning
• MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs
• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
• PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
• Test-Time Training Done Right
• VRAG: Learning World Models for Interactive Video Generation
• Effective Context in Neural Speech Models
• Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration
• Learning What to Remember: Test-Time Training via Context Distillation
• DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
• Hybrid SLC-MLC RRAM Mixed-Signal Processing-in-Memory Architecture for Transformer Acceleration via Gradient Redistribution
• Scale-invariant Attention
• CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
• KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
• Facts as First Class Objects: Knowledge Objects for Persistent LLM Memory
• Maglev: Sliding Recurrent Memory
• Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
• The Impossibility Triangle of Long-Context Modeling
• KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs

Discussions

• Leave No Context Behind: Efficient Infinite Context Transformers [hn, 39 points, 4 comments]
• Google: Efficient Infinite Context Transformers with Infini-Attention [hn, 4 points, 0 comments]
• Leave No Context Behind: Efficient Infinite Context Transformers [hn, 3 points, 0 comments]
• arxiv.org/abs/2404.07143 #LLM [bsky, 1 points, 0 comments]
• Will we see in 2025 the arrival of #LLMs with infinite + permanent compressive #memory that leave no context behind and humans may use as a sort of “second brain”? #AI #FuturOfWork arxiv.org/pdf/2404. [bsky, 0 points, 0 comments]

Related

• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
• Landmark Attention: Random-Access Infinite Context Length for Transformers
• A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
• Sliding Window Attention Training for Efficient Large Language Models
• Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling