Efficient uniform approximation using Random Vector Functional Link networks

2023/06/30 by Palina Salanevich, Salanevich, Palina, Olov Schavemaker +1 · 1 citation
Computer Science · #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2306.17501

Abstract

A Random Vector Functional Link (RVFL) network is a depth-2 neural network with random inner weights and biases. Only the outer weights of such an architecture are to be learned, so the learning process boils down to a linear optimization task, allowing one to sidestep the pitfalls of nonconvex optimization problems. In this paper, we prove that an RVFL with ReLU activation functions can approximate Lipschitz continuous functions in L_∞ norm. To the best of our knowledge, our result is the first approximation result in L_∞ norm using nice inner weights; namely, Gaussians. We give a nonasymptotic lower bound for the number of hidden-layer nodes to achieve a given accuracy with high probability, depending on, among other things, the Lipschitz constant of the target function, the desired accuracy, and the input dimension. Our method of proof is rooted in probability theory and harmonic analysis.

Cited by

• A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds

Related

• Approximation power of random neural networks
• Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces
• Quantified advantage of discontinuous weight selection in approximations with deep neural networks
• Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
• Stochastic choice of basis functions in adaptive function approximation and the functional-link net