2025/02/18 by Dhruv Rohatgi, Rohatgi, Dhruv, Adam Block +7 · 1 voice · 7 citations
Computer Science · #Natural Language Processing Techniques
paper · pdf · doi:10.48550/arxiv.2502.12465
Next-token prediction with the logarithmic loss is a cornerstone of autoregressive sequence modeling, but, in practice, suffers from error amplification, where errors in the model compound and generation quality degrades as sequence length H increases. From a theoretical perspective, this phenomenon should not appear in well-specified settings, and, indeed, a growing body of empirical work hypothesizes that misspecification, where the learner is not sufficiently expressive to represent the target distribution, may be the root cause. Under misspecification -- where the goal is to learn as well as the best-in-class model up to a multiplicative approximation factor C≥ 1 -- we confirm that C indeed grows with H for next-token prediction, lending theoretical support to this empirical hypothesis. We then ask whether this mode of error amplification is avoidable algorithmically, computationally, or information-theoretically, and uncover inherent computational-statistical tradeoffs. We show: (1) Information-theoretically, one can avoid error amplification and achieve C=O(1). (2) Next-token prediction can be made robust so as to achieve C= O(H), representing moderate error amplification, but this is an inherent barrier: any next-token prediction-style objective must suffer C=Ω(H). (3) For the natural testbed of autoregressive linear models, no computationally efficient algorithm can achieve sub-polynomial approximation factor C=e^(log H)1-Ω(1); however, at least for binary token spaces, one can smoothly trade compute for statistical power and improve on C=Ω(H) in sub-exponential time. Our results have consequences in the more general setting of imitation learning, where the widely-used behavior cloning algorithm generalizes next-token prediction.