2018/02/13 by Zhihao Jia, Jia, Zhihao, Sina Lin +5 · 2 citations
Computer Science · #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1802.04924
openalex publication_date 2018/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The past few years have witnessed growth in the computational requirements\nfor training deep convolutional neural networks. Current approaches parallelize\ntraining onto multiple devices by applying a single parallelization strategy\n(e.g., data or model parallelism) to all layers in a network. Although easy to\nreason about, these approaches result in suboptimal runtime performance in\nlarge-scale distributed training, since different layers in a network may\nprefer different parallelization strategies. In this paper, we propose\nlayer-wise parallelism that allows each layer in a network to use an individual\nparallelization strategy. We jointly optimize how each layer is parallelized by\nsolving a graph search problem. Our evaluation shows that layer-wise\nparallelism outperforms state-of-the-art approaches by increasing training\nthroughput, reducing communication costs, achieving better scalability to\nmultiple GPUs, while maintaining original network accuracy.\n