Survey on Large Scale Neural Network Training

2022/02/21 by Julia Gusak, Gusak, Julia, Daria Cherniuk +19 · 1 citation
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2202.10435

openalex publication_date 2022/02/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU device or can be trained using only a small per-GPU batch size. This survey provides a systematic overview of the approaches that enable more efficient DNNs training. We analyze techniques that save memory and make good use of computation and communication resources on architectures with a single or several GPUs. We summarize the main categories of strategies and compare strategies within and across categories. Along with approaches proposed in the literature, we discuss available implementations.

Cited by

• Slice or the Whole Pie? Utility Control for AI Models

Related

• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• FireCaffe: near-linear acceleration of deep neural network training on\n compute clusters
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design
• AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks