Approximate Softmax Functions for Energy-Efficient Deep Neural Networks

2022/12/01 by Ke Chen, Yue Gao, Haroon Waris +2 · 2 citations
Computer Science · Engineering · #Low-power high-performance VLSI design #Neural Networks and Applications #Parallel Computing and Optimization Techniques

paper · doi:10.1109/tvlsi.2022.3224011

openalex publication_date 2022/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Approximate computing has emerged as a new paradigm that provides power-efficient and high-performance arithmetic designs by relaxing the stringent requirement of accuracy. Nonlinear functions (such as softmax, rectified linear unit (ReLU), Tanh, and Sigmoid) are extensively used in deep neural networks (DNNs). However, they incur significant power dissipation due to the high circuit complexity. As DNNs are error-tolerant, the design of approximation-linear functions is possible and desired. In this article, the design of an approximate softmax function (AxSF) is proposed. AxSF is based on a double hybrid structure (DHS). AxSF divides the input of the softmax function into two parts for different processing methods. The most significant bits (MSBs) are processed with lookup tables (LUTs) and an exact restoring array divider (EXDr). Taylor’s expansion and a logarithmic divider are used for the less significant bits (LSBs). An improved DHS (IDHS) is also proposed to reduce the hardware complexity. In IDHS, a novel Booth multiplier is utilized for the hybrid scheme to improve the partial product generation and compression, while the truncated implementation is applied to the divider unit. The proposed DHS and IDHS are compared with existing softmax designs. The results show that the proposed approximate softmax design reduces hardware by 48% and delay by 54% while retaining a high accuracy.

Citations

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• The Pascal Visual Object Classes (VOC) Challenge
• Long Short-Term Memory
• A Fast Learning Algorithm for Deep Belief Nets
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Cognitive computing
• A High-Throughput and Power-Efficient FPGA Implementation of YOLO CNN for Object Detection

Cited by

• Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Acceleration
• QForce-RL: Quantized FPGA-Optimized Reinforcement Learning Compute Engine

Related

• A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks
• Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
• HEAM: High-Efficiency Approximate Multiplier Optimization for Deep Neural Networks
• Ax-BxP: Approximate Blocked Computation for Precision-Reconfigurable Deep Neural Network Acceleration
• Low Power Approximate Multiplier Architecture for Deep Neural Networks