PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters

2020/10/01 by Isabelly Rocha, Rocha, Isabelly, Nathaniel Morris +10 · 1 citation
Computer Science · #Advanced Data Storage Technologies #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Parallel #Parallel Computing and Optimization Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC

paper · pdf · doi:10.48550/arxiv.2010.00501

European Commission Project: LEGaTO - Low Energy Toolset for Heterogeneous Computing (EC-H2020-780681)

openalex publication_date 2020/10/01 · arxiv created 2020/10/02 · arxiv updated 2020/10/05 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

DNN learning jobs are common in today's clusters due to the advances in AI driven services such as machine translation and image recognition. The most critical phase of these jobs for model performance and learning cost is the tuning of hyperparameters. Existing approaches make use of techniques such as early stopping criteria to reduce the tuning impact on learning cost. However, these strategies do not consider the impact that certain hyperparameters and systems parameters have on training time. This paper presents PipeTune, a framework for DNN learning jobs that addresses the trade-offs between these two types of parameters. PipeTune takes advantage of the high parallelism and recurring characteristics of such jobs to minimize the learning cost via a pipelined simultaneous tuning of both hyper and system parameters. Our experimental evaluation using three different types of workloads indicates that PipeTune achieves up to 22.6% reduction and 1.7x speed up on tuning and training time, respectively. PipeTune not only improves performance but also lowers energy consumption up to 29%.

Citations

• On the difficulty of training Recurrent Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Learning to Forget: Continual Prediction with LSTM
• Hyperparameter Search in Machine Learning
• DeepID3: Face Recognition with Very Deep Neural Networks
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Variational Dropout Sparsifies Deep Neural Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Tune: A Research Platform for Distributed Model Selection and Training

Cited by

• GROOT: General-Purpose Automatic Parameter Tuning Across Layers, Domains, and Use Cases

Related

• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• HyperTune: Dynamic Hyperparameter Tuning For Efficient Distribution of DNN Training Over Heterogeneous Systems
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training
• Beyond Data and Model Parallelism for Deep Neural Networks