In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation

2025/02/12 by Cole, Frank, Zhao, Yuxuan, Lu, Yulong +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2502.08136

Abstract

This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems. Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an L²-testing loss uniformly defined across tasks. This result demonstrates that transformers with logarithmic depth can achieve error bounds comparable with those of the least-squares estimator. In contrast, our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers, which suggests a depth-separation phenomenon for transformers in the in-context learning of dynamical systems. Moreover, this second result uncovers a critical distinction in the approximation power of single-layer linear transformers when learning from IID versus non-IID data.

Cited by

• Pretrain-Test Task Alignment Governs Generalization in In-Context Learning

Related

• In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning
• On the Role of Depth and Looping for In-Context Learning with Task Diversity
• Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
• Transformers Meet In-Context Learning: A Universal Approximation Theory
• In-context Learning for Mixture of Linear Regressions: Existence, Generalization and Training Dynamics