A Comprehensive Study of Bugs in Modern Distributed Deep Learning Systems

2025/12/23 by Xiaoxue Ma, Ma, Xiaoxue, Wanwei Zhan +9
Computer Science · #Software System Performance and Reliability #Software Testing and Debugging Techniques #Adversarial Robustness in Machine Learning

paper · doi:10.48550/arxiv.2512.20345

Abstract

In today's data-driven era, deep learning is vital for processing massive datasets, yet single-device training is constrained by computational and memory limits. Distributed deep learning overcomes these challenges by leveraging multiple GPUs or machines in parallel. While general-purpose frameworks (e.g., TensorFlow and PyTorch) provide distributed capabilities, these are often add-on features that demand significant manual effort for advanced parallelism, underscoring the need for specialized frameworks. This study conducts the first large-scale empirical analysis of practitioner challenges in dedicated distributed frameworks. We examine 849 real-world issues from DeepSpeed, Megatron-LM, and Colossal-AI and construct a taxonomy of 34 bug symptoms, 28 root causes, and 6 fix patterns. Crucially, we establish explicit mappings between symptoms, causes, and fixes across distributed training stages, enabling a systematic understanding of how issues emerge and are resolved. Our results show that 45.1% of bug symptoms are unique to distributed frameworks, with setup failures, memory issues, and performance anomalies being the most prevalent. Moreover, 95% of issues in the communication setup stage occur exclusively in distributed contexts. We also find over 60% of cases can be resolved through version and dependency management, and distributed feature, API, and communication tuning. Based on these findings, we provide actionable implications.

Citations

• Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study
• Distributed Inference and Fine-tuning of Large Language Models Over The Internet
• An Empirical Study on Bugs Inside PyTorch: A Replication Study
• DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
• On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub
• Toward Understanding Deep Learning Framework Bugs
• Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow
• Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective
• Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
• Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
• A Survey on Deep Learning for Software Engineering
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU\n Clusters through Integration of Pipelined Model Parallelism and Data\n Parallelism
• Repairing Deep Neural Networks: Fix Patterns and Challenges
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Taxonomy of Real Faults in Deep Learning Systems
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Horovod: fast and easy distributed deep learning in TensorFlow
• TensorFlow: A system for large-scale machine learning
• ImageNet classification with deep convolutional neural networks

Related

• Towards Understanding Bugs in Distributed Training and Inference Frameworks for Large Language Models
• Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective
• Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey
• A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
• TTrace: Lightweight Error Checking and Diagnosis for Distributed Training