Demystifying Parallel and Distributed Deep Learning

2019/08/30 by Tal Ben-Nun, Tal Ben‐Nun, Torsten Hoefler · 26 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Ferroelectric and Negative Capacitance Devices #Stochastic Gradient Optimization Techniques

paper · doi:10.1145/3320060

openalex publication_date 2019/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

Citations

• Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• cuDNN: Efficient Primitives for Deep Learning
• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• A Stochastic Approximation Method
• Scheduling multithreaded computations by work stealing
• Long Short-Term Memory
• Finding Structure in Time
• Gradient-based learning applied to document recognition
• Optimal Distributed Online Prediction using Mini-Batches
• Do Deep Nets Really Need to be Deep?
• A Fast Learning Algorithm for Deep Belief Nets
• Backpropagation Applied to Handwritten Zip Code Recognition
• ImageNet classification with deep convolutional neural networks
• A Survey on Transfer Learning
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• MapReduce
• 8-Bit Approximations for Parallelism in Deep Learning
• Sparse Tensor Algebra as a Parallel Programming Model
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• A guide to convolution arithmetic for deep learning
• Training Deep Nets with Sublinear Memory Cost
• End to End Learning for Self-Driving Cars
• Ternary Weight Networks
• Memory-Efficient Backpropagation Through Time
• Optimization Methods for Large-Scale Machine Learning
• Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Neural Architecture Search with Reinforcement Learning
• Generative Adversarial Parallelization
• On the Performance of Network Parallel Training in Artificial Neural Networks
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• DeepArchitect: Automatically Designing and Training Deep Architectures
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• YellowFin and the Art of Momentum Tuning
• One Model To Learn Them All
• Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Large Batch Training of Convolutional Networks
• Meta Learning Shared Hierarchies
• Don't Decay the Learning Rate, Increase the Batch Size
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Progressive Neural Architecture Search
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Efficient Neural Architecture Search via Parameter Sharing
• Efficient Sparse-Winograd Convolutional Neural Networks
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Communication-Optimal Convolutional Neural Nets
• DARTS: Differentiable Architecture Search
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Xception: Deep Learning with Depthwise Separable Convolutions
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Numerical optimization
• Learning Multiple Layers of Features from Tiny Images
• The Byzantine Generals Problem

Cited by

• CAPS: Fine-Tuning CCA Timing
• Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models
• An Additively Preconditioned Trust Region Strategy for Machine Learning
• Bandwidth-Aware Network Topology Optimization for Decentralized Learning
• AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
• Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training
• Memory Efficient and Staleness Free Pipeline Parallel DNN Training Framework with Improved Convergence Speed
• Resilient AI Supercomputer Networking using MRC and SRv6
• Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approach
• Energy Consumption in Parallel Neural Network Training
• Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
• CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
• Empirically-Calibrated H100 Node Power Models for Reducing Uncertainty in AI Training Energy Estimation
• SDR-RDMA: Software-Defined Reliability Architecture for Planetary Scale RDMA Communication
• Performance Characterization of Distributed Deep Learning Strategies: A Quantitative Evaluation of DDP, FSDP, and Parameter Server Architectures on GPU Clusters
• ATLAHS: An Application-centric Network Simulator Toolchain for AI, HPC, and Distributed Storage
• Quantum ring all-reduce: communication and privacy advantages for distributed learning
• GDEV-AI: A Generalized Evaluation of Deep Learning Inference Scaling and Architectural Saturation
• Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication
• Parallel Training of Deep Networks with Local Updates
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence
• LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
• Improving Accuracy and Speeding Up Document Image Classification Through Parallel Systems
• Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs
• Flexible Communication Avoiding Matrix Multiplication on FPGA with High-Level Synthesis

Related

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability
• A Linear Algebraic Approach to Model Parallelism in Deep Learning
• A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
• Beyond Data and Model Parallelism for Deep Neural Networks