2024/11/13 by Dennis Rochau, Rochau, Dennis, Robin Chan +3
Computer Science · Mathematics · #Advanced Computational Techniques in Science and Engineering #FOS: Computer and information sciences #FOS: Mathematics #Functional Analysis (math.FA) #Image and Signal Denoising Methods #Iterative Methods for Nonlinear Equations #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.2411.08735
openalex publication_date 2024/11/13 · openalex created_date 2024/11/16 · openalex updated_date 2026/07/28
We prove several universal approximation results at minimal or near-minimal width for approximation of Lp(ℝdx, ℝdy) and C0(ℝdx, ℝdy) on compact sets. Our approach uses a unified coding scheme that yields explicit constructions relying only on standard analytic tools. We show that feedforward neural networks with two leaky ReLU activations σα, σ-α achieve the optimal width max\dx, dy\ for Lp approximation, while a single leaky ReLU σα achieves width max\2, dx, dy\, providing an alternative proof of the results of Cai et al. (2023). By generalizing to stepped leaky ReLU activations, we extend these results to uniform approximation of continuous functions while identifying sets of activation functions compatible with gradient-based training. Since our constructions pass through an intermediate dimension of one, they imply that autoencoders with a one-dimensional feature space are universal approximators. We further show that squashable activations combined with FLOOR achieve width max\3, dx, dy\ for uniform approximation. We also establish a lower bound of max\dx, dy\ + 1 for networks when all activations are continuous and monotone and dy ≤ 2dx. Moreover, we extend our results to invertible LU-decomposable networks, proving distributional universal approximation for LU-Net normalizing flows and providing a constructive proof of the classical theorem of Brenier and Gangbo on Lp approximation by diffeomorphisms.