Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

2025/07/02 by Shikai Qiu, Qiu, Shikai, Lechao Xiao +8 · 3 voices · 11 citations
Computer Science · Physics and Astronomy · #Artificial neural network #Dynamics (music) #Hyperparameter #Learning curve #Model Reduction and Neural Networks #Neural Networks and Applications #Noise (video) #Scaling #cs.LG

paper · pdf · doi:10.48550/arxiv.2507.02119

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/02 · openalex created_date 2025/10/20 · openalex updated_date 2026/08/05

Abstract

What scaling limits govern neural network training dynamics when model size and training time grow in tandem? We show that despite the complex interactions between architecture, training algorithms, and data, compute-optimally trained models exhibit a remarkably precise universality. Specifically, loss curves from models of varying sizes collapse onto a single universal curve when training compute and loss are normalized to unity at the end of training. With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor of individual loss curves across random seeds, a phenomenon we term supercollapse. We observe supercollapse across learning rate schedules, datasets, and architectures, including transformers trained on next-token prediction, and find it breaks down when hyperparameters are scaled suboptimally, providing a precise and practical indicator of good scaling. We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws, and analyzing a simple yet surprisingly effective model of SGD noise dynamics that accurately predicts loss curves across various learning rate schedules and quantitatively explains the origin of supercollapse.

Citations

• The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
• Scaling Exponents Across Parameterizations and Optimizers
• Infinite Limits of Multi-head Transformer Dynamics
• Training Compute-Optimal Large Language Models
• Deep Learning Scaling is Predictable, Empirically

Cited by

• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• Decoupled DiLoCo for Resilient Distributed Pre-training
• Per-example gradients: a new frontier for understanding and improving optimizers
• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Efficient Hyperparameter Tuning via Trajectory Invariance Principle
• Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
• Towards joint scaling laws with optimal batch size schedules
• Explaining Data Mixing Scaling Laws
• Anomalous diffusion in convergence to effective ergodicity

Discussions

• Cool paper claiming that the behavior of loss curves for compute optimally trained models is scale invariant(?) and you can use this to detect whether your distributed training setup is improperly con [bsky, 31 points, 1 comments]
• Our new ICML paper discovers scaling collapse: through a simple affine transformation, whole training loss curves across model sizes with optimally scaled hypers collapse to a single universal curve! [bsky, 30 points, 3 comments]
• arxiv.org/abs/2507.02119 [bsky, 0 points, 0 comments]

Related

• A Dynamical Model of Neural Scaling Laws
• Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
• The Optimization Landscape of SGD Across the Feature Learning Strength
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• A Tale of Tails: Model Collapse as a Change of Scaling Laws