ReLU² Wins: Discovering Efficient Activation Functions for Sparse LLMs

2024/02/06 by Zhengyan Zhang, Yixin Song, Zhang, Zhengyan +17 · 2 voices · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2402.03804

Abstract

Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neurons. While traditional approaches focus on ReLU-based LLMs, leveraging zeros in activation values, we broaden the scope of sparse LLMs beyond zero activation values. We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold, demonstrating that non-ReLU LLMs also exhibit sparse activation. To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU². The results indicate that models employing ReLU² excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs. We will release the code to facilitate future research.

Cited by

• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
• Universal Properties of Activation Sparsity in Modern Large Language Models
• DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\
• SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment

Discussions

• this is the first one i am finding but there've been a few of these. any time you a/b test relu^2 against the gated activation family it seems to win arxiv.org/abs/2402.03804 [bsky, 4 points, 0 comments]
• paper: arxiv.org/pdf/2402.03804 [bsky, 1 points, 0 comments]

Related

• ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
• SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference
• Achieving Sparse Activation in Small Language Models
• CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification