An MPI-Based Python Framework for Distributed Training with Keras

2017/12/16 by Anderson, Dustin, Vlimant, Jean-Roch, Spiropulu, Maria · 2 citations
#Distributed #FOS: Computer and information sciences #Machine Learning (stat.ML) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.1712.05878

Abstract

We present a lightweight Python framework for distributed training of neural networks on multiple GPUs or CPUs. The framework is built on the popular Keras machine learning library. The Message Passing Interface (MPI) protocol is used to coordinate the training process, and the system is well suited for job submission at supercomputing sites. We detail the software's features, describe its use, and demonstrate its performance on systems of varying sizes on a benchmark problem drawn from high-energy physics research.

Cited by

• Identification of tau leptons using a convolutional neural network with domain adaptation
• From ANN to BNN: Inferring Reionization Parameters using Uncertainty-aware Emulators of 21-cm Summaries

Related

• Theano-MPI: a Theano-based Distributed Training Framework
• BAGUA: Scaling up Distributed Learning with System Relaxations
• Applications of Deep Neural Networks with Keras
• Data-parallel distributed training of very large models beyond GPU capacity
• Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity