Approximating Continuous Functions by ReLU Nets of Minimal Width

2017/10/31 by Boris Hanin, Mark Sellke, Hanin, Boris +1 · 9 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Advanced Numerical Analysis Techniques #Combinatorics (math.CO) #Computational Complexity (cs.CC) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.1710.11278

openalex publication_date 2017/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed d_in≥ 1, what is the minimal width w so that neural nets with ReLU activations, input dimension d_in, hidden layer widths at most w, and arbitrary depth can approximate any continuous, real-valued function of d_in variables arbitrarily well? It turns out that this minimal width is exactly equal to d_in+1. That is, if all the hidden layer widths are bounded by d_in, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions, and, on the other hand, any continuous function on the d_in-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly d_in+1. Our construction in fact shows that any continuous function f:[0,1]^d_in→\mathbb R^d_out can be approximated by a net of width d_in+d_out. We obtain quantitative depth estimates for such an approximation in terms of the modulus of continuity of f.

Citations

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Opening the Black Box of Deep Neural Networks via Information

Cited by

• Upper Approximation Bounds for Neural Oscillators
• Minimum Width of Deep Narrow Networks for Universal Approximation
• Dimensionality reduction and width of deep neural networks based on topological degree theory
• An in-depth look at approximation via deep and narrow neural networks
• Model reduction of parametric ordinary differential equations via autoencoders: structure-preserving latent dynamics and convergence analysis
• Symbolic Feedforward Networks for Probabilistic Finite Automata: Exact Simulation and Learnability
• DeepOPF: A Deep Neural Network Approach for Security-Constrained DC Optimal Power Flow
• Algebraically-Informed Deep Networks (AIDN): A Deep Learning Approach to\n Represent Algebraic Structures
• Error Estimation and Correction from within Neural Network Differential\n Equation Solvers

Related

• Sharp Representation Theorems for ReLU Networks with Precise Dependence\n on Depth
• Optimal approximation of continuous functions by very deep ReLU networks
• Neural Networks with Small Weights and Depth-Separation Barriers
• Approximation capabilities of neural networks on unbounded domains
• On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem