Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

2023/05/29 by Liang Li, Yifei Duan, Li, Li'ang +5 · 2 citations
Computer Science · #Neural Networks and Applications #Image and Signal Denoising Methods #Digital Filter Design and Implementation

paper · pdf · doi:10.48550/arxiv.2305.18460

Abstract

The study of universal approximation properties (UAP) for neural networks (NN) has a long history. When the network width is unlimited, only a single hidden layer is sufficient for UAP. In contrast, when the depth is unlimited, the width for UAP needs to be not less than the critical width w^*_min=max(d_x,d_y), where d_x and d_y are the dimensions of the input and output, respectively. Recently, \citecai2022achieve shows that a leaky-ReLU NN with this critical width can achieve UAP for L^p functions on a compact domain K, i.e., the UAP for L^p(K,ℝ^d_y). This paper examines a uniform UAP for the function class C(K,ℝ^d_y) and gives the exact minimum width of the leaky-ReLU NN as w_min=max(d_x,d_y)+Δ(d_x, d_y), where Δ(d_x, d_y) is the additional dimensions for approximating continuous functions with diffeomorphisms via embedding. To obtain this result, we propose a novel lift-flow-discretization approach that shows that the uniform UAP has a deep connection with topological theory.

Cited by

• Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding
• Model reduction of parametric ordinary differential equations via autoencoders: structure-preserving latent dynamics and convergence analysis

Related

• Achieve the Minimum Width of Neural Networks for Universal Approximation
• New advances in universal approximation with neural networks of minimal width
• Minimum width for universal approximation using ReLU networks on compact domain
• Minimum Width of Deep Narrow Networks for Universal Approximation
• Minimum Width for Universal Approximation