DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models

2025/05/28 by Alex Iacob, Iacob, Alex, Lorenzo Sani +18 · 1 citation
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques #Stochastic Gradient Optimization Techniques

paper · doi:10.48550/arxiv.2505.22549

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training.

Citations

• Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• ADOPT: Modified Adam Can Converge with Any β₂ with the Optimal Rate
• How Does Critical Batch Size Scale in Pre-training?
• DEPT: Decoupled Embeddings for Pre-training Language Models
• Convergence of Distributed Adaptive Optimization with Local Updates
• The AdEMAMix Optimizer: Better, Faster, Older
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• The Future of Large Language Model Pre-training is Federated
• Asynchronous Local-SGD Training for Language Modeling
• Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
• DiLoCo: Distributed Low-Communication Training of Language Models
• Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
• Stable and low-precision training for large-scale vision-language models
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• Symbolic Discovery of Optimization Algorithms
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
• Adam Can Converge Without Any Modification On Update Rules
• On Distributed Adaptive Optimization with Gradient Compression
• PaLM: Scaling Language Modeling with Pathways
• Training Compute-Optimal Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• An Improved Analysis of Stochastic Gradient Descent with Momentum
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Advances and Open Problems in Federated Learning
• Lower Bounds for Non-Convex Stochastic Optimization
• PIQA: Reasoning about Physical Commonsense in Natural Language
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD\n for Distributed Non-Convex Optimization
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Local SGD Converges Fast and Communicates Little
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Horovod: fast and easy distributed deep learning in TensorFlow
• On the Convergence of Adam and Beyond
• Don't Decay the Learning Rate, Increase the Batch Size
• Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
• On the difficulty of training Recurrent Neural Networks
• Photon: Federated LLM Pre-Training

Cited by

• MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates

Related

• MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
• DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
• Distributed Low-Communication Training with Decoupled Momentum Optimization
• DiLoCo: Distributed Low-Communication Training of Language Models
• LoCo: Low-Bit Communication Adaptor for Large-scale Model Training