Low-Memory Neural Network Training: A Technical Report

2019/04/24 by Nimit S. Sohoni, Sohoni, Nimit S., Christopher R. Aberger +7 · 2 voices · 38 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Bottleneck #Computer science #Domain Adaptation and Few-Shot Learning #Embedded system #High memory #Inference #Machine learning #Memory model #Parallel computing #Shared memory #Transformer #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1904.10631

published in arXiv (Cornell University) (Cornell University) · Version notes: Copyedits and citation fixes

openalex publication_date 2019/04/24 · arxiv created 2022/04/08 · arxiv updated 2022/04/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Memory is increasingly often the bottleneck when training neural network models. Despite this, techniques to lower the overall memory requirements of training have been less widely studied compared to the extensive literature on reducing the memory requirements of inference. In this paper we study a fundamental question: How much memory is actually needed to train a neural network? To answer this question, we profile the overall memory usage of training on two representative deep learning benchmarks -- the WideResNet model for image classification and the DynamicConv Transformer model for machine translation -- and comprehensively evaluate four standard techniques for reducing the training memory requirements: (1) imposing sparsity on the model, (2) using low precision, (3) microbatching, and (4) gradient checkpointing. We explore how each of these techniques in isolation affects both the peak memory usage of training and the quality of the end model, and explore the memory, accuracy, and computation tradeoffs incurred when combining these techniques. Using appropriate combinations of these techniques, we show that it is possible to the reduce the memory required to train a WideResNet-28-2 on CIFAR-10 by up to 60.7x with a 0.4% loss in accuracy, and reduce the memory required to train a DynamicConv model on IWSLT'14 German to English translation by up to 8.7x with a BLEU score drop of 0.15.

Citations

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Cited by

• Enabling Binary Neural Network Training on the Edge
• Reformer: The Efficient Transformer
• On improving deep learning generalization with adaptive sparse connectivity
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• On the Downstream Performance of Compressed Word Embeddings
• How Low Can We Go: Trading Memory for Error in Low-Precision Training
• Geminet: Learning the Duality-based Iterative Process for Lightweight Traffic Engineering in Changing Topologies
• GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
• Enabling Incremental Training with Forward Pass for Edge Devices
• Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
• Hydra: A System for Large Multi-Model Deep Learning
• Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions

Discussions

• Low-Memory Neural Network Training: A Technical Report [hn, 58 points, 3 comments]
• Low-Memory Neural Network Training: A Technical Report [hn, 8 points, 0 comments]

Related

• Low-Resolution Neural Networks
• Accelerating Neural Network Training: A Brief Review
• Small Batch Sizes Improve Training of Low-Resource Neural MT
• A Challenging Benchmark for Low-Resource Learning
• TBD: Benchmarking and Analyzing Deep Neural Network Training