DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

2026/03/19 by Maoyang Xiang, Bo Wang
#cs.LG

paper · pdf · doi:10.1145/3770743.3804274

Abstract

Non-linear activation functions play a pivotal role in on-device inference and training, as they not only consume substantial hardware resources but also impose a significant impact on system performance and energy efficiency. In this work, we propose Distribution-Aware Piecewise Activation (DAPA), a differentiable and hardware-friendly activation function for Transformer architectures by exploiting the distribution of pre-activation data. DAPA employs a non-uniform piecewise approximation that allocates finer segments to high-probability regions of the distribution, improving generalizability over prior piecewise linear methods. The resulting approximation is further quantized using Distribution-Weighted Mean Square Error to reduce latency and resource utilization for hardware deployment. Our HLS implementation demonstrates that DAPA speeds up GELU computation by 16× and decreases DSP utilization by 16× while maintaining comparable or better performance across vision Transformers and GPT-2 models.

Citations

• HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline
• PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers
• Reusing Softmax Hardware Unit for GELU Computation in Transformers
• Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation
• SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers
• Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
• Training data-efficient image transformers & distillation through attention
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• ImageNet Large Scale Visual Recognition Challenge

Related

• Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference
• CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference
• Optimizing the Deployment of Tiny Transformers on Low-Power MCUs
• TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference
• Efficient Resource-Constrained Training of Vision Transformers via Subspace Optimization