Repeat After Me: Transformers are Better than State Space Models at Copying

2024/02/01 by Samy Jelassi, Jelassi, Samy, David Brandfonbrener +5 · 35 citations
Decision Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Scientific Computing and Data Management #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2402.01032

openalex publication_date 2024/02/01 · openalex created_date 2024/02/06 · openalex updated_date 2026/07/28

Abstract

Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized state space models" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.

Cited by

• Raven: High-Recall Sequence Modeling with Sparse Memory Routing
• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
• Characterizing Mamba's Selective Memory using Auto-Encoders
• Provable Long-Range Benefits of Next-Token Prediction
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
• Selective Rotary Position Embedding
• Attention and Compression is all you need for Controllably Efficient Language Models
• FlashEVA: Accelerating LLM inference via Efficient Attention
• Quantitative Bounds for Length Generalization in Transformers
• MossNet: Mixture of State-Space Experts is a Multi-Head Attention
• Comparing Transformers and Hybrid Models at the Token Level
• The Bayesian Geometry of Transformer Attention
• Sequences of Logits Reveal the Low Rank Structure of Language Models
• StreetMath: Study of LLMs' Approximation Behaviors
• Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
• To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
• Design Principles for Sequence Models via Coefficient Dynamics
• gLSTM: Mitigating Over-Squashing by Increasing Storage Capacity
• A Comparative Analysis of Contextual Representation Flow in State-Space and Transformer Architectures
• Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis
• Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
• TTT3R: 3D Reconstruction as Test-Time Training
• Enabling Physical AI through Biological Principles
• Mamba Modulation: On the Length Generalization of Mamba
• How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
• Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
• Towards Understanding Visual Grounding in Visual Language Models
• Fast attention mechanisms: a tale of parallelism
• Customizing the Inductive Biases of Softmax Attention using Structured Matrices
• PLaMo 2 Technical Report
• WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration
• Revisiting associative recall in modern recurrent models

Related

• Sovereign Byte Firewall: Comparing Transformer and Mamba-2 Backbones for Tokenization-Free Zero-Day Network Anomaly Detection
• The Expressive Capacity of State Space Models: A Formal Language Perspective
• The Illusion of State in State-Space Models
• Block-State Transformers
• TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding