Towards joint scaling laws with optimal batch size schedules

2026/07/30 by Jiaxiang Li, Zhiqi Bu, Shiyun Xu
Computer Science · Mathematics · #cs.LG #cs.AI #math.OC

paper · pdf

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.

Citations

• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
• Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
• Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
• Qwen3 Technical Report
• Don't be lazy: CompleteP enables compute-efficient deep transformers
• The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
• Scaling Law for Language Models Training Considering Batch Size
• How Does Critical Batch Size Scale in Pre-training?
• Scaling Optimal LR Across Token Horizons
• The Llama 3 Herd of Models
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Orca-Math: Unlocking the potential of SLMs in Grade School Math
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• Optimal Linear Decay Learning Rate Schedules and Further Refinements
• Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
• GLM-130B: An Open Bilingual Pre-trained Model
• PaLM: Scaling Language Modeling with Pathways
• Training Compute-Optimal Large Language Models
• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• An Empirical Model of Large-Batch Training
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Don't Decay the Learning Rate, Increase the Batch Size
• Attention Is All You Need
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Coupling Adaptive Batch Sizes with Learning Rates
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining

Related

• Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit
• Optimal Growth Schedules for Batch Size and Learning Rate in SGD that Reduce SFO Complexity
• The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
• Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling