2025/08/02 by S. Kudo, Kudo, Shiko
Computer Science · Engineering · Physics and Astronomy · #41A30 #65D15 (Secondary) #68T07 (Primary) 42A10 #FOS: Computer and information sciences #FOS: Mathematics #Ferroelectric and Negative Capacitance Devices #G.1.2 #G.1.6 #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Numerical Analysis (math.NA)
paper · pdf · doi:10.48550/arxiv.2508.01175
openalex publication_date 2025/08/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The dominant paradigm in modern neural networks relies on simple, monotonically-increasing activation functions like ReLU. While effective, this paradigm necessitates large, massively-parameterized models to approximate complex functions. In this paper, we introduce the Periodic Linear Unit (PLU), a learnable sine-wave based activation with periodic non-monotonicity. PLU is designed for maximum expressive power and numerical stability, achieved through its formulation and a paired innovation we term Repulsive Reparameterization, which prevents the activation from collapsing into a non-expressive linear function. We demonstrate that a minimal MLP with only two PLU neurons can solve the spiral classification task, a feat impossible for equivalent networks using standard activations. This suggests a paradigm shift from networks as piecewise Taylor-like approximators to powerful Fourier-like function synthesizers, achieving exponential gains in parameter efficiency by placing intelligence in the neuron itself.