Softermax: Hardware/Software Co-Design of an Efficient Softmax for\n Transformers

2021/03/16 by Jacob R. Stevens, Rangharajan Venkatesan, Stevens, Jacob R. +7 · 6 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Hardware Architecture (cs.AR) #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2103.09301

openalex publication_date 2021/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformers have transformed the field of natural language processing. This\nperformance is largely attributed to the use of stacked self-attention layers,\neach of which consists of matrix multiplies as well as softmax operations. As a\nresult, unlike other neural networks, the softmax operation accounts for a\nsignificant fraction of the total run-time of Transformers. To address this, we\npropose Softermax, a hardware-friendly softmax design. Softermax consists of\nbase replacement, low-precision softmax computations, and an online\nnormalization calculation. We show Softermax results in 2.35x the energy\nefficiency at 0.90x the size of a comparable baseline, with negligible impact\non network accuracy.\n

Cited by

• Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency
• IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
• QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operations
• SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors

Related

• SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
• Reusing Softmax Hardware Unit for GELU Computation in Transformers
• Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference
• VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers
• Neural Architecture Search on Efficient Transformers and Beyond