Theory-to-Practice Gap for Neural Networks and Neural Operators

2025/03/23 by Philipp Grohs, Samuel Lanthaler, Grohs, Philipp +3
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Markov Chains and Monte Carlo Methods #Stochastic Gradient Optimization Techniques #cs.LG #math.FA

paper · pdf · doi:10.48550/arxiv.2503.18219

openalex publication_date 2025/03/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work studies the sampling complexity of learning with ReLU neural networks and neural operators. For mappings belonging to relevant approximation spaces, we derive upper bounds on the best-possible convergence rate of any learning algorithm, with respect to the number of samples. In the finite-dimensional case, these bounds imply a gap between the parametric and sampling complexities of learning, known as the theory-to-practice gap. In this work, a unified treatment of the theory-to-practice gap is achieved in a general L^p-setting, while at the same time improving available bounds in the literature. Furthermore, based on these results the theory-to-practice gap is extended to the infinite-dimensional setting of operator learning. Our results apply to Deep Operator Networks and integral kernel-based neural operators, including the Fourier neural operator. We show that the best-possible convergence rate in a Bochner L^p-norm is bounded by rates of order 1/p.

Citations

• Probabilistic weather forecasting with machine learning
• Deep Neural Network Approximation Theory

Related

• Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spaces
• Sampling Complexity of Deep Approximation Spaces
• Data Complexity Estimates for Operator Learning
• Statistical Learning Theory for Neural Operators
• Operator Learning: Algorithms and Analysis