An in-depth look at approximation via deep and narrow neural networks

2025/10/08 by Joris Dommel, Dommel, Joris, Sven‐Ake Wegner +1
Computer Science · #41A30 #68T07 #FOS: Computer and information sciences #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2510.07202

openalex publication_date 2025/10/08 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

In 2017, Hanin and Sellke showed that the class of arbitrarily deep, real-valued, feed-forward and ReLU-activated networks of width w forms a dense subset of the space of continuous functions on Rⁿ, with respect to the topology of uniform convergence on compact sets, if and only if w>n holds. To show the necessity, a concrete counterexample function f:Rⁿ->R was used. In this note we actually approximate this very f by neural networks in the two cases w=n and w=n+1 around the aforementioned threshold. We study how the approximation quality behaves if we vary the depth and what effect (spoiler alert: dying neurons) cause that behavior.

Citations

• Improved weight initialization for deep and narrow feedforward neural network
• Universality of Gradient Descent Neural Network Training
• Sharp Representation Theorems for ReLU Networks with Precise Dependence\n on Depth
• Optimal Function Approximation with Relu Neural Networks
• On decision regions of narrow deep neural networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Adam: A Method for Stochastic Optimization
• Multilayer feedforward networks are universal approximators

Related

• On the Expressive Power of Neural Networks
• Approximation spaces of deep neural networks
• Universal Approximation with Deep Narrow Networks
• A Numerical Investigation of the Minimum Width of a Neural Network
• Why Deep Neural Networks for Function Approximation?