Architectural Implications of Neural Network Inference for High Data-Rate, Low-Latency Scientific Applications

2024/03/13 by Olivia Weng, Alexander Redding, Weng, Olivia +7
Computer Science · Engineering · #Advanced Neural Network Applications #CCD and CMOS Imaging Sensors #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2403.08980

openalex publication_date 2024/03/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

With more scientific fields relying on neural networks (NNs) to process data incoming at extreme throughputs and latencies, it is crucial to develop NNs with all their parameters stored on-chip. In many of these applications, there is not enough time to go off-chip and retrieve weights. Even more so, off-chip memory such as DRAM does not have the bandwidth required to process these NNs as fast as the data is being produced (e.g., every 25 ns). As such, these extreme latency and bandwidth requirements have architectural implications for the hardware intended to run these NNs: 1) all NN parameters must fit on-chip, and 2) codesigning custom/reconfigurable logic is often required to meet these latency and bandwidth constraints. In our work, we show that many scientific NN applications must run fully on chip, in the extreme case requiring a custom chip to meet such stringent constraints.

Related

• ONNX-to-Hardware Design Flow for Adaptive Neural-Network Inference on FPGAs
• EDEN: Enabling Energy-Efficient, High-Performance Deep Neural Network Inference Using Approximate DRAM
• Exploiting Neural-Network Statistics for Low-Power DNN Inference
• ONNX-to-Hardware Design Flow for the Generation of Adaptive Neural-Network Accelerators on FPGAs
• Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloud