Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

2020/06/28 by Tao Luo, Haizhao Yang, Luo, Tao +1 · 50 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Advanced Numerical Analysis Techniques #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #FOS: Computer and information sciences #FOS: Mathematics #Generalization #Gradient descent #Least-squares function approximation #Machine Learning (cs.LG) #Mathematical analysis #Mathematical optimization #Mathematics #Model Reduction and Neural Networks #Neural Networks and Applications #Norm (philosophy) #Numerical Analysis (math.NA) #Optimization and Control (math.OC) #Optimization problem #Parametrization (atmospheric modeling) #Partial differential equation #cs.LG #cs.NA #math.NA #math.OC

paper · pdf · doi:10.48550/arxiv.2006.15733

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2020/06/28 · openalex created_date 2020/07/02 · arxiv created 2020/12/10 · arxiv updated 2020/12/14 · openalex updated_date 2026/08/06

Abstract

The problem of solving partial differential equations (PDEs) can be formulated into a least-squares minimization problem, where neural networks are used to parametrize PDE solutions. A global minimizer corresponds to a neural network that solves the given PDE. In this paper, we show that the gradient descent method can identify a global minimizer of the least-squares optimization for solving second-order linear PDEs with two-layer neural networks under the assumption of over-parametrization. We also analyze the generalization error of the least-squares optimization for second-order linear PDEs and two-layer neural networks, when the right-hand-side function of the PDE is in a Barron-type space and the least-squares optimization is regularized with a Barron-type norm, without the over-parametrization assumption.

Citations

• Densely Connected Convolutional Networks
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Gradient Descent Finds Global Minima of Deep Neural Networks
• A type of generalization error induced by initialization in deep neural networks
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth

Cited by

• A mixed residual method for biharmonic equations in spectral Barron spaces
• What Can One Expect When Solving PDEs Using Shallow Neural Networks?
• Multi-Scale Finite Expression Method for PDEs with Oscillatory Solutions on Complex Domains
• Error analysis for the deep Kolmogorov method
• Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation
• SSBE-PINN: A Sobolev Boundary Scheme Boosting Stability and Accuracy in Elliptic/Parabolic PDE Learning
• A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs
• Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equations
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Quantum Circuit Encodings of Polynomial Chaos Expansions
• Convergence Guarantees for Gradient-Based Training of Neural PDE Solvers: From Linear to Nonlinear PDEs
• Solving Nonlinear PDEs with Sparse Radial Basis Function Networks
• Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables
• Error Analysis of Deep Ritz Methods for Elliptic Equations

Related

• Error estimates of residual minimization using neural networks for linear PDEs
• Optimally weighted loss functions for solving PDEs with Neural Networks
• First Order System Least Squares Neural Networks
• Greedy Training Algorithms for Neural Networks and Applications to PDEs
• PDE-constrained Models with Neural Network Terms: Optimization and Global Convergence