Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis

2025/10/21 by Jian Lü, Lu, Jian, Xiaohuang Huang +1 · 1 citation
Computer Science · Physics and Astronomy · #41A46 #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.18388

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness of the function being approximated within the Barron function space. We examine the approximation rates of ReLU^k activation functions, proving that the optimal rate cannot be achieved under ℓ¹-bounded coefficients or insufficient smoothness conditions. We also establish optimal approximation rates in various norms for functions in Barron spaces and Sobolev spaces, confirming the curse of dimensionality. Our results clarify the limits of shallow neural networks' approximation capabilities and offer insights into the selection of activation functions and network structures.

Citations

• Characterization of the Variation Spaces Corresponding to Shallow Neural Networks
• High-Order Approximation Rates for Shallow Neural Networks with Cosine and ReLU^k Activation Functions
• Optimal Stable Nonlinear Approximation
• Sharp Representation Theorems for ReLU Networks with Precise Dependence\n on Depth
• Approximation by Combinations of ReLU and Squared ReLU Ridge Functions\n with \ℓ¹ and \ℓ⁰ Controls
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Estimation of Linear Functionals on Sobolev Spaces with Application to Fourier Transforms and Spline Interpolation

Cited by

• Neural Network Approximation of Solutions to Fractional Parabolic Partial Differential Equations

Related

• High-Order Approximation Rates for Shallow Neural Networks with Cosine and ReLU^k Activation Functions
• Sharp Lower Bounds on the Approximation Rate of Shallow Neural Networks
• Approximation Rates for Shallow ReLU^k Neural Networks on Sobolev Spaces via the Radon Transform
• On the Expressive Power of Neural Networks
• Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks