2020/05/16 by Zhigang Liu, Liu, Zhi-Gang, Paul N. Whatmough +3
Computer Science · Engineering · Mathematics · #Advanced Memory and Neural Computing #Distributed #FOS: Computer and information sciences #FOS: Electrical engineering #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Parallel #Parallel Computing and Optimization Techniques #Signal Processing (eess.SP) #Tensor decomposition and applications #and Cluster Computing (cs.DC) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2005.08098
openalex publication_date 2020/05/16 · openalex created_date 2020/05/21 · openalex updated_date 2026/07/28
Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this work, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. Firstly, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08x and 1.36x respectively, compared to the conventional SA at iso-throughput with INT8 operands. Secondly, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14x and 1.97x improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.