Loop the Loopies!

2026/07/17 by Zitian Gao, Yilong Chen, Yihao Xiao +4 · 2 voices
#cs.CL #cs.AI

paper · pdf

Abstract

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

Citations

• CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
• Scaling Latent Reasoning via Looped Language Models
• Hierarchical Reasoning Model
• Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
• Transformer Layers as Painters
• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
• Transformers learn in-context by gradient descent
• Training Compute-Optimal Large Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Discussions

• Loop the Loopies [hn, 2 points, 0 comments]
• [30/30] 172 Upvotes, 2 Comments, 3 Posts, arXiv:2607.16051 🆕Loop the Loopies! Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou [bsky, 0 points, 1 comments]

Related

• Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners
• Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
• Looped Transformers are Better at Learning Learning Algorithms
• Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
• Continual Pre-training of MoEs: How robust is your router?