Continuous Autoregressive Language Models

2025/10/31 by Chenze Shao, Darren Li, Dongbai Li +6 · 16 voices
Computer Science · Social Sciences · #Topic Modeling #Natural Language Processing Techniques #Computational and Text Analysis Methods

paper · pdf · doi:10.48550/arxiv.2510.27688

Abstract

The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a new design axis for LLM scaling: increasing the semantic bandwidth of each generative step. To this end, we introduce Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9% accuracy. This allows us to model language as a sequence of continuous vectors instead of discrete tokens, which reduces the number of generative steps by a factor of K. The paradigm shift necessitates a new modeling toolkit; therefore, we develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling in the continuous domain. Experiments show that CALM significantly improves the performance-compute trade-off, achieving the performance of strong discrete baselines at a significantly lower computational cost. More importantly, these findings establish next-vector prediction as a powerful and scalable pathway towards ultra-efficient language models. Code: https://github.com/shaochenze/calm. Project: https://shaochenze.github.io/blog/2025/CALM.

Citations

• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Byte Latent Transformer: Patches Scale Better Than Tokens
• Better & Faster Large Language Models via Multi-token Prediction
• On the Dangers of Stochastic Parrots
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Taming Transformers for High-Resolution Image Synthesis
• Energy and Policy Considerations for Deep Learning in NLP
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Cited by

• Context-level Language Modeling by Learning Predictive Context Embeddings

Discussions

• Continuous Autoregressive Language Models [hn, 115 points, 10 comments]
• Continuous Autoregressive Language Models [hn, 3 points, 1 comments]
• TL;DR Instead of predicting one token at a time, CALM predicts continuous vectors that represent multiple tokens at once. Research paper : arxiv.org/abs/2510.27688 [bsky, 2 points, 0 comments]
• Continuous Autoregressive Language Models [hn, 2 points, 0 comments]
• Very interesting article on how to compress Large Language Models Llm tokens: arxiv.org/abs/2510.27688 [bsky, 1 points, 0 comments]
• 📌This is fascinating. Nice to see a major lab agrees with me that the discrete token paradigm is fundamentally broken. arxiv.org/abs/2510.27688 [bsky, 1 points, 0 comments]
• CALM by Tencent Team uses continuous vector prediction, improving efficiency and performance while reducing computational costs significantly in LLM design. arxiv.org/abs/2510.27688 [bsky, 1 points, 0 comments]
• arxiv.org/abs/2510.27688 [bsky, 1 points, 0 comments]
• arxiv.org/abs/2510.27688 [bsky, 0 points, 0 comments]
• CALMの仕組み——4トークンの塊を128次元のベクトル1個に圧縮し、その「ベクトル列」を予測する。1ステップで4トークン分の意味を出すゆえ、計算量は減る。良い研究じゃ。 じゃが、論文自身がこう書いておる。 「同じパラメータ数なら、標準Transformerの方が性能は高い」💎 CALMは「賢さ」で負け、「計算効率」で勝っておるだけ。「全LLM時代遅れ」とは、論文自身が言うておらぬのじゃよ。 h [bsky, 0 points, 1 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3m5j4vpmgbq42 [bsky, 0 points, 0 comments]
• Continuous Autoregressive Language Models arxiv.org/abs/2510.27688 [bsky, 0 points, 0 comments]
• This could be a breakthrough arxiv.org/abs/2510.27688 [bsky, 0 points, 0 comments]
• Current transformers tokenize on words (which is not entirely true, more like part of words), but CALM compresses a chunk of tokens into a single continuous vector, while maintaining 99.9% of the qual [bsky, 0 points, 0 comments]
• #AI @cnrs.fr @cea.fr @inriaparisnlp.bsky.social arxiv.org/abs/2510.27688 [bsky, 0 points, 0 comments]
• Continuous Autoregressive Language Models https://arxiv.org/abs/2510.27688 [bsky, 0 points, 0 comments]

Related

• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
• Exploring the Hidden Capacity of LLMs for One-Step Text Generation
• Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time