Addressing the Memory Bottleneck in AI Model Training

2020/03/11 by Ojika, David, Patel, Bhavesh, Reina, G. Anthony +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2003.08732

Abstract

Using medical imaging as case-study, we demonstrate how Intel-optimized TensorFlow on an x86-based server equipped with 2nd Generation Intel Xeon Scalable Processors with large system memory allows for the training of memory-intensive AI/deep-learning models in a scale-up server configuration. We believe our work represents the first training of a deep neural network having large memory footprint (~ 1 TB) on a single-node server. We recommend this configuration to scientists and researchers who wish to develop large, state-of-the-art AI models but are currently limited by memory.

Related

• Systems for Parallel and Distributed Large-Model Deep Learning Training
• System and Design Technology Co-optimization of SOT-MRAM for High-Performance AI Accelerator Memory System
• Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
• Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models
• Exploring shared memory architectures for end-to-end gigapixel deep learning