2024/02/09 by Liangyu Zhao, Zhao, Liangyu, Saeed Maleki +9 · 1 citation
Computer Science · #Distributed #FOS: Computer and information sciences #Interconnection Networks and Systems #Machine Learning (cs.LG) #Networking and Internet Architecture (cs.NI) #Opportunistic and Delay-Tolerant Networks #Parallel #Software-Defined Networks and 5G #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2402.06787
openalex publication_date 2024/02/09 · openalex created_date 2024/02/14 · openalex updated_date 2026/07/28
As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's heterogeneous and diverse network fabrics. We present ForestColl, a tool that generates throughput-optimal schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretical optimality. Its schedule generation runs in polynomial time and is highly scalable. ForestColl supports any network fabric, including both switching fabrics and direct accelerator connections. We evaluated ForestColl on AMD MI250 and NVIDIA DGX A100 & H100 clusters. ForestColl showed significant improvements over the vendors' own optimized communication libraries across various settings and in LLM training. ForestColl also outperformed other state-of-the-art schedule generation techniques with both more efficient generated schedules and substantially faster generation speed.