Understanding and mitigating gradient pathologies in physics-informed neural networks

2020/01/13 by Sifan Wang, Wang, Sifan, Yujun Teng +3 · 38 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Nuclear Engineering Thermal-Hydraulics #Nuclear reactor physics and engineering #Numerical Analysis (math.NA) #cs.LG #cs.NA #math.NA #stat.ML

paper · pdf · doi:10.48550/arxiv.2001.04536

28 Pages, 18 figures

arxiv created 2020/01/13 · openalex publication_date 2020/01/13 · arxiv updated 2020/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \urlhttps://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs.

Citations

• On the Construction and Comparison of Difference Schemes
• Understanding Deep Convolutional Networks
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Covariant Compositional Networks For Learning Graphs
• Neural Proximal Gradient Descent for Compressive Imaging
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Competitive Gradient Descent
• Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data

Cited by

• More Consistent Accuracy PINN via Alternating Easy-Hard Training
• RRaPINNs: Residual Risk-Aware Physics Informed Neural Networks
• Data-integrated neural networks for solving partial differential equations
• Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics
• Self-adaptive weighting and sampling for physics-informed neural networks
• Quantifying constraint hierarchies in Bayesian PINNs via per-constraint Hessian decomposition
• Optimal Transport Based Refinement of Physics-Informed Neural Networks
• Examining the robustness of Physics-Informed Neural Networks to noise for Inverse Problems
• Physics-Informed Neural Networks for Heat Transfer Problems
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Learning the solution operator of parametric partial differential equations with physics-informed DeepONets
• Choose a Transformer: Fourier or Galerkin
• Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions
• FEDONet : Fourier-Embedded DeepONet for Spectrally Accurate Operator Learning
• NVIDIA SimNet^TM: an AI-accelerated multi-physics simulation framework
• TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
• Geometric flow regularization in latent spaces for smooth dynamics with the efficient variations of curvature
• Polynomial-Spline Neural Networks with Exact Integrals
• Estimation of Hemodynamic Parameters via Physics Informed Neural Networks including Hematocrit Dependent Rheology
• Bridging ocean wave physics and deep learning: Physics-informed neural operators for nonlinear wavefield reconstruction in real-time
• Reconstructing a dynamical system and forecasting time series by self-consistent deep learning
• AdjointNet: Constraining machine learning models with physics-based codes
• On the Correspondence between Gaussian Processes and Geometric Harmonics
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• PINNeik: Eikonal solution using physics-informed neural networks
• PDE-aware Optimizer for Physics-informed Neural Networks
• Room impulse response reconstruction with physics-informed deep learning
• Spectral Bias in Variational Quantum Machine Learning
• Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networks
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• Over-PINNs: Enhancing Physics-Informed Neural Networks via Higher-Order Partial Derivative Overdetermination of PDEs
• Physics-informed machine learning
• Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks
• Physics-informed Reduced Order Modeling of Time-dependent PDEs via Differentiable Solvers
• A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
• A Physics-Informed Spatiotemporal Deep Learning Framework for Turbulent Systems
• A deep neural network approach on solving the linear transport model under diffusive scaling
• Machine Learning for Vortex Induced Vibration in Turbulent Flow

Related

• A deep learning theory for neural networks grounded in physics
• Improved physics-informed neural network in mitigating gradient related failures
• Critical Investigation of Failure Modes in Physics-informed Neural Networks
• Enforcing hidden physics in physics-informed neural networks
• Characterizing and Mitigating the Difficulty in Training Physics-informed Artificial Neural Networks under Pointwise Constraints