Minimum Width of Deep Narrow Networks for Universal Approximation

2025/11/10 by Yang, Xiao-Song, Zhou, Qi, Zhou, Xuan
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · doi:10.48550/arxiv.2511.06837

openalex publication_date 2025/11/10 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

Determining the minimum width of fully connected neural networks has become a fundamental problem in recent theoretical studies of deep neural networks. In this paper, we study the lower bounds and upper bounds of the minimum width required for fully connected neural networks in order to have universal approximation capability, which is important in network design and training. We show that w_min≤max(2d_x+1, d_y) also holds true for networks with ELU, SELU activation functions, and the upper bound of this inequality is attained when d_y=2d_x, where d_x, d_y denote the input and output dimensions, respectively. Besides, we show that d_x+1≤ w_min≤ d_x+d_y for networks with LeakyReLU, ELU, CELU, SELU, Softplus activation functions, by proving that ReLU activation function can be approximated by these activation functions. In addition, in the case that the activation function is injective or can be uniformly approximated by a sequence of injective functions (e.g., ReLU), we present a new proof of the inequality w_min≥ d_y+1_d_x

Citations

• When narrower is better: the narrow width limit of Bayesian parallel branching neural networks
• Mathematical theory of deep learning
• Improved weight initialization for deep and narrow feedforward neural network
• Minimum width for universal approximation using ReLU networks on compact domain
• Achieve the Minimum Width of Neural Networks for Universal Approximation
• On the Optimal Memorization Power of ReLU Neural Networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Provable Memorization via Deep Neural Networks using Sub-linear Parameters
• Minimum Width for Universal Approximation
• Topology of deep neural networks
• Universal Approximation with Deep Narrow Networks
• Deep, Skinny Neural Networks are not Universal Approximators
• Optimal approximation of continuous functions by very deep ReLU networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• The Zero Set of a Real Analytic Function
• Deep Residual Learning for Image Recognition
• ImageNet classification with deep convolutional neural networks
• Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Related

• Minimum width for universal approximation using ReLU networks on compact domain
• Minimum Width for Universal Approximation
• Achieve the Minimum Width of Neural Networks for Universal Approximation
• Universal Approximation with Deep Narrow Networks
• A Numerical Investigation of the Minimum Width of a Neural Network