2025/08/21 by Amir Sarfi, Sarfi, Amir, Benjamin Thérien +6 · 4 voices · 4 citations
Computer Science · #Advanced Neural Network Applications #Speech Recognition and Synthesis #Topic Modeling #cs.LG
paper · pdf · doi:10.48550/arxiv.2508.15706
openalex publication_date 2025/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwidth-constrained settings, such as across datacenters and over the internet. While these local-update methods achieve communication reduction through reduced synchronization, they still require communicating dense model-sized pseudo-gradients, resulting in a communication bottleneck even for cross-datacenter links. While quantization is often applied to reduce the pseudo-gradient's size, in the context of LLM pre-training, existing approaches have not been able to leverage sparsification without incurring significant performance degradation. In this work, we introduce SparseLoCo, a communication-efficient training algorithm for LLMs that can effectively leverage Top-k sparsification and 2-bit quantization to reach extreme sparsity in the communicated pseudo-gradient, as high as 97-99%, while achieving lower final loss than dense DiLoCo. In our empirical study of language model pre-training, we demonstrate that SparseLoCo's improvements over DiLoCo in performance and pseudo-gradient compression are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing the number of workers, and increasing communication intervals.