Effective Approaches to Batch Parallelization for Dynamic Neural Network Architectures

2017/07/08 by Joseph Suárez, Suarez, Joseph, Clare Zhu +1
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1707.02402

openalex publication_date 2017/07/08 · openalex created_date 2017/07/21 · openalex updated_date 2026/07/28

Abstract

We present a simple dynamic batching approach applicable to a large class of dynamic architectures that consistently yields speedups of over 10x. We provide performance bounds when the architecture is not known a priori and a stronger bound in the special case where the architecture is a predetermined balanced tree. We evaluate our approach on Johnson et al.'s recent visual question answering (VQA) result of his CLEVR dataset by Inferring and Executing Programs (IEP). We also evaluate on sparsely gated mixture of experts layers and achieve speedups of up to 1000x over the naive implementation.

Citations

• Long Short-Term Memory
• Deep Learning with Dynamic Computation Graphs
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Related

• ED-Batch: Efficient Automatic Batching of Dynamic Neural Networks via Learned Finite State Machines
• On-the-fly Operation Batching in Dynamic Computation Graphs
• An optimal scheduling architecture for accelerating batch algorithms on Neural Network processor architectures
• TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with Removed Staleness
• Training Large Neural Networks with Constant Memory using a New Execution Algorithm