Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep\n Neural Networks

2017/05/03 by Minsoo Rhu, Rhu, Minsoo, Mike O’Connor +7 · 4 citations
Computer Science · Engineering · #Advanced Neural Network Applications #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1705.01626

openalex publication_date 2017/05/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Popular deep learning frameworks require users to fine-tune their memory\nusage so that the training data of a deep neural network (DNN) fits within the\nGPU physical memory. Prior work tries to address this restriction by\nvirtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be\nutilized for memory allocations. Despite its merits, virtualizing memory can\nincur significant performance overheads when the time needed to copy data back\nand forth from CPU memory is higher than the latency to perform the\ncomputations required for DNN forward and backward propagation. We introduce a\nhigh-performance virtualization strategy based on a "compressing DMA engine"\n(cDMA) that drastically reduces the size of the data structures that are\ntargeted for CPU-side allocations. The cDMA engine offers an average 2.6x\n(maximum 13.8x) compression ratio by exploiting the sparsity inherent in\noffloaded data, improving the performance of virtualized DNNs by an average 32%\n(maximum 61%).\n

Cited by

• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
• Chameleon: Taming Dynamic Operator Sequences for Memory-Intensive LLM Training
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training

Related

• vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Less Memory Means smaller GPUs: Backpropagation with Compressed Activations