Higher-order Quasi-Monte Carlo Training of Deep Neural Networks

2020/09/06 by Marcello Longo, Siddhartha Mishra, Longo, M. +5 · 1 citation
Computer Science · Decision Sciences · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Numerical Analysis (math.NA) #Numerical Methods and Algorithms #Probabilistic and Robust Engineering Design

paper · pdf · doi:10.48550/arxiv.2009.02713

openalex publication_date 2020/09/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a novel algorithmic approach and an error analysis leveraging Quasi-Monte Carlo points for training deep neural network (DNN) surrogates of Data-to-Observable (DtO) maps in engineering design. Our analysis reveals higher-order consistent, deterministic choices of training points in the input data space for deep and shallow Neural Networks with holomorphic activation functions such as tanh. These novel training points are proved to facilitate higher-order decay (in terms of the number of training samples) of the underlying generalization error, with consistency error bounds that are free from the curse of dimensionality in the input data space, provided that DNN weights in hidden layers satisfy certain summability conditions. We present numerical experiments for DtO maps from elliptic and parabolic PDEs with uncertain inputs that confirm the theoretical analysis.

Citations

• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

Cited by

• Neural Low-Discrepancy Sequences

Related

• Regularity and tailored regularization of Deep Neural Networks, with application to parametric PDEs in uncertainty quantification
• Multilevel Monte Carlo learning
• Quasi-Monte Carlo sampling for machine-learning partial differential equations
• Robust DNN Surrogate Models with Uncertainty Quantification via Adversarial Training
• Practical Quasi-Newton Methods for Training Deep Neural Networks