2021/11/09 by Daniel Nichols, Nichols, Daniel, Siddharth Singh +6 · 1 citation
Computer Science · Medicine · Neuroscience · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #COVID-19 diagnosis using AI #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC) #cs.AI #cs.DC #cs.LG
paper · pdf · doi:10.48550/arxiv.2111.04949
openalex publication_date 2021/11/09 · arxiv created 2022/07/01 · arxiv updated 2022/07/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The field of deep learning has witnessed a remarkable shift towards extremely compute- and memory-intensive neural networks. These newer larger models have enabled researchers to advance state-of-the-art tools across a variety of fields. This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators. In this paper, we discuss and compare current state-of-the-art frameworks for large scale distributed deep learning. First, we survey current practices in distributed learning and identify the different types of parallelism used. Then, we present empirical results comparing their performance on large image and language training tasks. Additionally, we address their statistical efficiency and memory consumption behavior. Based on our results, we discuss algorithmic and implementation portions of each framework which hinder performance.