Parle: parallelizing stochastic gradient descent

2017/07/03 by Pratik Chaudhari, Chaudhari, Pratik, Carlo Baldassi +9 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Distributed #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1707.00424

openalex publication_date 2017/07/03 · arxiv created 2017/09/10 · arxiv updated 2017/09/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

Citations

• Deep Image: Scaling up Image Recognition
• Gradient-based learning applied to document recognition
• Striving for Simplicity: The All Convolutional Net
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Staleness-aware Async-SGD for Distributed Deep Learning
• Wide Residual Networks
• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Flat Minima
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Learning Multiple Layers of Features from Tiny Images

Cited by

• LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization

Related

• Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• Distributed Deep Learning using Stochastic Gradient Staleness
• Asynchronous Parallel Stochastic Gradient Descent - A Numeric Core for Scalable Distributed Machine Learning Algorithms
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling