Approximation capabilities of neural networks on unbounded domains

2019/10/21 by Wang, Ming-Xi, Qu, Yang · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1910.09293

Abstract

In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]ⁿ. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function expresses no nonzero integrable function defined on the Euclidean plane. Together with a recent result that the deep ReLU network can arbitrarily well approximate any integrable function on Euclidean spaces, we provide a new perspective on the advantage of multiple hidden layers in the context of ReLU networks. Lastly, we prove that the ReLU network with depth 3 is a universal approximator in L^p(Rⁿ).

Cited by

• LUNA: Linear Universal Neural Attention with Generalization Guarantees

Related

• On the Expressive Power of Neural Networks
• Approximation in L^p(μ) with deep ReLU neural networks
• Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces
• Universal approximation results for neural networks with non-polynomial activation function over non-compact domains
• Error bounds for approximations with deep ReLU networks