Compressive Transformers for Long-Range Sequence Modelling

2019/11/13 by Rae, Jack W., Potapenko, Anna, Jayakumar, Siddhant M. +1 · 67 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1911.05507

Abstract

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Cited by

• TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
• Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
• Beyond Context: Large Language Models Failure to Grasp Users Intent
• Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifolds
• Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments
• PAACE: A Plan-Aware Automated Agent Context Engineering Framework
• GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• Associative Memory using Attribute-Specific Neuron Groups-1: Learning between Multiple Cue Balls
• Visualizing LLM Latent Space Geometry Through Dimensionality Reduction
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• Sentence-Anchored Gist Compression for Long-Context LLMs
• Attention and Compression is all you need for Controllably Efficient Language Models
• BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models
• ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation
• Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
• SpecAttn: Speculating Sparse Attention
• Gaperon: A Peppered English-French Generative Language Model Suite
• MemSFT: Mitigating Alignment Tax with an External Parametric Memory
• ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
• Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows
• ARC-Encoder: learning compressed text representations for large language models
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
• Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
• MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
• Predicting Task Performance with Context-aware Scaling Laws
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• NOSA: Native and Offloadable Sparse Attention
• RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference
• MeSH: Memory-as-State-Highways for Recursive Transformers
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems
• ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning
• Associative Memory Model with Neural Networks: Memorizing multiple images with one neuron
• Revisiting Long-context Modeling from Context Denoising Perspective
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• MemMamba: Rethinking Memory Patterns in State Space Model
• Lossless Compression: A New Benchmark for Time Series Model Evaluation
• Mamba Modulation: On the Length Generalization of Mamba
• Recall Before You Rank: Similarity-Guided Top-K Reuse for Efficient Long-Context Attention
• Limitations of Normalization in Attention Mechanism
• PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• Positional Encoding via Token-Aware Phase Attention
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
• Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
• CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling
• Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
• HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
• Vision encoders should be image size agnostic and task driven
• REFRAG: Rethinking RAG based Decoding
• DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
• SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
• Hydra: A Modular Architecture for Efficient Long-Context Reasoning
• Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
• Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
• Exploring Cross-Utterance Speech Contexts for Conformer-Transducer Speech Recognition Systems
• Retrospective Sparse Attention for Efficient Long-Context Generation
• DeCAL Tokenwise Compression
• LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training
• Goal-Based Vision-Language Driving

Related

• DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence
• State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era
• Scaling Transformer to 1M tokens and beyond with RMT
• Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers
• Memory-efficient Stochastic methods for Memory-based Transformers