A Systolic Array Architecture for Nonlinear Activation Functions and Softmax Computation using Chebyshev Polynomials

2026/08/05 by Benedikt Schaible, Anirudh Suresh Bharadwaj, Ulf Schlichtmann +1
Computer Science · #cs.AR

paper · pdf

Accepted for IEEE COINS 2026

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Neural Network Accelerators have gained popularity in recent years due to their greater efficiency than CPU-based platforms. Often, these accelerators utilize different hardware units for univariate activation functions, such as tanh, and the multivariate softmax, thereby missing opportunities for resource sharing between them. In this paper, we describe a novel systolic array-based activation unit architecture that supports multiple univariate activation functions as well as the softmax function. By utilizing Chebyshev polynomial approximations, our activation function unit achieves up to 71% lower mean absolute error for tanh compared to a CORDIC baseline, while using 4.6% less area and 5.1% less power. Our softmax approximation enables a 44.6% and 79.0% lower KL divergence compared to CORDIC and a piecewise-linear approximation, respectively.

Citations

• ONE-SA: Enabling Nonlinear Operations in Systolic Arrays for Efficient and Flexible Neural Network Inference
• Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
• Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
• Design Space Exploration of Neural Network Activation Function Circuits
• In-Datacenter Performance Analysis of a Tensor Processing Unit
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• XXI. A new method of solving numerical equations of all orders, by continuous approximation