Parallel Layer Normalization for Universal Approximation

2025/05/19 by Yunhao Ni, Yuxin Guo, Ni, Yunhao +12
Computer Science · #Advanced Graph Neural Networks #Approximation error #Approximation theory #Artificial neural network #Deep neural networks #FOS: Computer and information sciences #Layer (electronics) #Linear approximation #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Norm (philosophy) #Normalization (sociology) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.13142

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

This paper studies the approximation capabilities of neural networks that combine layer normalization (LN) with linear layers. We prove that networks consisting of two linear layers with parallel layer normalizations (PLNs) inserted between them (referred to as PLN-Nets) achieve universal approximation, whereas architectures that use only standard LN exhibit strictly limited expressive power.We further analyze approximation rates of shallow and deep PLN-Nets under the L^∞ norm as well as in Sobolev norms. Our analysis extends beyond LN to RMSNorm, and from standard MLPs to position-wise feed-forward networks, the core building blocks used in RNNs and Transformers.Finally, we provide empirical experiments to explore other possible potentials of PLN-Nets.

Citations

• Minimum width for universal approximation using squashable activation functions
• On the Nonlinearity of Layer Normalization
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Retentive Network: A Successor to Transformer for Large Language Models
• Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Minimum Width for Universal Approximation
• Are Transformers universal approximators of sequence-to-sequence functions?
• Root Mean Square Layer Normalization
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Understanding Batch Normalization
• How Does Batch Normalization Help Optimization?
• Group Normalization
• Attention Is All You Need
• Error bounds for approximations with deep ReLU networks
• Layer Normalization
• Multi30K: Multilingual English-German Image Descriptions
• Deep Residual Learning for Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• On the Number of Linear Regions of Deep Neural Networks
• ImageNet classification with deep convolutional neural networks
• Long Short-Term Memory

Related

• An elementary proof of a universal approximation theorem
• A Survey on Universal Approximation Theorems
• Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks
• Achieve the Minimum Width of Neural Networks for Universal Approximation
• New advances in universal approximation with neural networks of minimal width