GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

2021/02/15 by Heesu Kim, Kim, Heesu, Hanmin Park +15 · 3 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2102.07511

openalex publication_date 2021/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

In this paper, we present GradPIM, a processing-in-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.

Citations

• Gradient-based learning applied to document recognition
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Cited by

• CryptoSRAM: Enabling High-Throughput Cryptography on MCUs via In-SRAM Computing
• Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
• Accelerating Triangle Counting with Real Processing-in-Memory Systems

Related

• PIM-DRAM: Accelerating Machine Learning Workloads using Processing in Commodity DRAM
• A New MRAM-based Process In-Memory Accelerator for Efficient Neural Network Training with Floating Point Precision
• Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloud
• Hybrid In-memory Computing Architecture for the Training of Deep Neural Networks
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks