Exploring Scaling Laws for Local SGD in Large Language Model Training

2024/09/20 by Qiaozhi He, Xiaomin Zhuang, He, Qiaozhi +4 · 1 voice
Computer Science · Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2409.13198

openalex publication_date 2024/09/20 · arxiv published 2024/09/20 · arxiv updated 2024/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive experiments, we show that local SGD achieves competitive results compared to conventional methods, given equivalent model parameters, datasets, and computational resources. Furthermore, we explore the application of local SGD in various practical scenarios, including multi-cluster setups and edge computing environments. Our findings elucidate the necessary conditions for effective multi-cluster LLM training and examine the potential and limitations of leveraging edge computing resources in the LLM training process. This demonstrates its viability as an alternative to single large-cluster training.

Discussions

• With DiLoCo, you can now be as flops/token efficient as DP training while using two orders of magnitude less bandwidth. Follow ups (arxiv.org/abs/2409.13198, arxiv.org/abs/2405.10853, arxiv.org/abs/24 [bsky, 4 points, 1 comments]

Related

• EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
• HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
• Scaling Law for Language Models Training Considering Batch Size
• DiLoCo: Distributed Low-Communication Training of Language Models
• Taming LLMs by Scaling Learning Rates with Gradient Grouping