Scaling Latent Reasoning via Looped Language Models

2025/10/29 by Rui-Jie Zhu, Zixuan Wang, Zhu, Rui-Jie +63 · 14 voices · 21 citations
#cs.CL

paper · pdf · doi:10.48550/arxiv.2510.25741

Abstract

Modern LLMs are trained to "think" primarily via explicit text generation, such as chain-of-thought (CoT), which defers reasoning to post-training and under-leverages pre-training data. We present and open-source Ouro, named after the recursive Ouroboros, a family of pre-trained Looped Language Models (LoopLM) that instead build reasoning into the pre-training phase through (i) iterative computation in latent space, (ii) an entropy-regularized objective for learned depth allocation, and (iii) scaling to 7.7T tokens. Ouro 1.4B and 2.6B models enjoy superior performance that match the results of up to 12B SOTA LLMs across a wide range of benchmarks. Through controlled experiments, we show this advantage stems not from increased knowledge capacity, but from superior knowledge manipulation capabilities. We also show that LoopLM yields reasoning traces more aligned with final outputs than explicit CoT. We hope our results show the potential of LoopLM as a novel scaling direction in the reasoning era. Our model is available here: http://ouro-llm.github.io.

Cited by

• Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
• Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
• LatentMT: Machine Translation with Latent Reasoning
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary
• Loop the Loopies!
• Training Continuous Chain of Thought Models: A Tale of Two Regimes
• T²MLR: Transformer with Temporal Middle-Layer Recurrence
• Energy-guided Recursive Model
• Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
• Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
• Diversity or Precision? A Deep Dive into Next Token Prediction
• DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning
• VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse
• AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
• Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
• Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle
• HRM-Text: Efficient Pretraining Beyond Scaling
• NeuroAI and Beyond: Bridging Between Advances in Neuroscience and ArtificialIntelligence
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
• Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
• A Formal Comparison Between Chain-of-Thought and Latent Thought
• The Topological Trouble With Transformers
• Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling

Discussions

• Scaling Latent Reasoning via Looped Language Models [hn, 84 points, 15 comments]
• 2/2 Bisherige Fortschritte bei LLMs basieren stark auf Skalierung: mehr Parameter, mehr Daten, mehr Rechenleistung. ​Das Problem ist, dass im Internet langsam die hochwertigen, von Menschen erstellten [bsky, 11 points, 1 comments]
• Scaling Latent Reasoning via Looped Language Models https://arxiv.org/abs/2510.25741 [comments] [41 points] [bsky, 4 points, 0 comments]
• "Scaling latent reasoning via looped language models". Here, the output of transformers is looped inside the neural network without any conversion to tokens. Every transformer section of the model can [bsky, 2 points, 0 comments]
• Seems related: arxiv.org/abs/2510.25741 [bsky, 2 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models https://arxiv.org/abs/2510.25741 [bsky, 0 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models https://arxiv.org/abs/2510.25741 [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3mbkmlhwkphn2 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2510.25741 [bsky, 0 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models https:// arxiv.org/abs/2510.25741 # arxiv [mastodon, 0 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models https://arxiv.org/abs/2510.25741 (https://news.ycombinator.com/item?id=46481849) [bsky, 0 points, 0 comments]
• Like I get why people are obsessed with disproving LLM intelligence But even foundation models aren’t just language anymore and many researchers are exploring beyond the “plain ass transformer” space [bsky, 0 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models https://arxiv.org/abs/2510.25741 (https://news.ycombinator.com/item?id=46481849) [bsky, 0 points, 0 comments]
• Scaling Latent Reasoning via Looped Language Models [bsky, 0 points, 0 comments]

Related

• Reasoning with Latent Thoughts: On the Power of Looped Transformers
• Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
• SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs
• Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
• Demystifying Long Chain-of-Thought Reasoning in LLMs