When and why PINNs fail to train: A neural tangent kernel perspective

2021/10/12 by Sifan Wang, Xinling Yu, Paris Perdikaris · 1,267 citations
Decision Sciences · Engineering · Mathematics · Physics and Astronomy · #Algorithm #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #Convergence (economics) #Discrete mathematics #Geometry #Gradient descent #Kernel (algebra) #Limit (mathematics) #Mathematical analysis #Mathematical optimization #Mathematics #Model Reduction and Neural Networks #Nuclear reactor physics and engineering #Probabilistic and Robust Engineering Design #Tangent

paper · open access · doi:10.1016/j.jcp.2021.110768

published in Journal of Computational Physics 449, 110768 (Elsevier BV)

openalex publication_date 2021/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Citations

• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations

Cited by

• Groundwater inverse modeling: Physics-informed neural network with disentangled constraints and errors
• Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers
• Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes
• An extreme learning machine-based method for computational PDEs in higher dimensions
• AT-PINN-HC: A refined time-sequential method incorporating hard-constraint strategies for predicting structural behavior under dynamic loads
• A framework based on symbolic regression coupled with eXtended Physics-Informed Neural Networks for gray-box learning of equations of motion from data
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• Respecting causality for training physics-informed neural networks
• LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks
• Double screening in the training dynamics of variational physics-informed neural networks for heterogeneous coupled parabolic systems
• A Structure-Adaptive Random Feature Method for High-Dimensional Elliptic PDEs
• PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs
• Low-Rank Evolutionary Deep Neural Networks via Adaptive Tangent-Space Reduction
• Trainable Spline Representations for Physics-Informed Learning
• Evolutionary Algorithm-Guided LLMs for Physics-Informed Neural Network Design
• Probabilistic Physics-Informed Neural Networks for Estimating Heterogeneous Elastic Properties from Low-Resolution and Noisy Displacement Data
• Material-agnostic temperature field prediction for metal additive manufacturing via a parametric PINN framework
• Reliability-Aware Hard--Soft Physics-Informed Neural Networks for Robust Learning of Challenging Partial Differential Equations
• DBAW-PIKAN: Dynamic Balance Adaptive Weight Kolmogorov-Arnold Neural Network for Solving Partial Differential Equations
• Finite-PINN: A Physics-Informed Neural Network with Finite Geometric Encoding for Solid Mechanics
• A Spectral-based Physics-informed Finite Operator Learning for Prediction of Mechanical Behavior of Microstructures
• Variational Boosting for Physics-Informed Neural Networks
• Perturbative-NeuSA: A Structured Spectral Framework for Time-Dependent PDEs
• Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
• Forward and Inverse Mantle Convection with Neural Operators
• Overcoming Spectral Bias via Cross-Attention
• Gradient Starvation: A Learning Proclivity in Neural Networks
• KD-PINN: Knowledge-Distilled PINNs for ultra-low-latency real-time neural PDE solvers
• On Parameter Identification in Three-Dimensional Elasticity and Discretisation with Physics-Informed Neural Networks
• Physics-Informed Neural Networks for Joint Source and Parameter Estimation in Advection-Diffusion Equations
• Multi-Distributed Sampling Method to Optimize Physical-Informed Neural Networks for Solving Optical Solitons
• Hankel-FNO: Fast Underwater Acoustic Charting Via Physics-Encoded Fourier Neural Operator
• An operator preconditioning perspective on training in physics-informed machine learning
• DAE-HardNet: A Physics Constrained Neural Network Enforcing Differential-Algebraic Hard Constraints
• A Multi-Phase Dual-PINN Framework: Soft Boundary-Interior Specialization via Distance-Weighted Priors
• Discontinuity-aware physics-informed neural network for phase-field method in three-phase flow with phase change
• Active learning with physics-informed neural networks for optimal sensor placement in deep tunneling through transversely isotropic elastic rocks
• A Global Spacetime Optimization Approach to the Real-Space Time-Dependent Schrödinger Equation
• RRaPINNs: Residual Risk-Aware Physics Informed Neural Networks
• Self-Scalable Tanh (Stan): Multi-Scale Solutions for Physics-Informed Neural Networks
• Convergence and Sketching-Based Efficient Computation of Neural Tangent Kernel Weights in Physics-Based Loss
• Enforcing hidden physics in physics-informed neural networks
• Data-integrated neural networks for solving partial differential equations
• Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics
• Neural-Initialized Newton: Accelerating Nonlinear Finite Elements via Operator Learning
• Learning Biomolecular Motion: The Physics-Informed Machine Learning Paradigm
• Physics-Informed Neural Networks for Real-Time Gas Crossover Prediction in PEM Electrolyzers: First Application with Multi-Membrane Validation
• Precipitation nowcasting of satellite data using physically-aligned neural networks
• Self-adaptive weighting and sampling for physics-informed neural networks
• A Two-stage Adaptive Lifting PINN Framework for Solving Viscous Approximations to Hyperbolic Conservation Laws
• Physics-Informed Neural Networks for Speech Production
• PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passes
• LieSolver: A PDE-constrained solver for IBVPs using Lie symmetries
• Physics-Informed Broad Learning System: An Efficient Backpropagation-Free Framework for Solving Partial Differential Equations
• IS-PINN: Integrating mathematical principles to solve inhomogeneous wave equations via physics-informed neural networks
• EvoPINN: Agentic Discovery of Executable Algorithms for Physics-Informed Neural Networks
• An inversion method for two-dimensional PT-symmetric potential functions using an ADI-Newton-GMRES framework
• TI-PINN: Topological-identifier physics-informed neural networks for forward and inverse problems of discontinuous coefficient differential equations
• Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Framework
• Enhancing Training of Physics-Informed Neural Networks Using Domain Decomposition–Based Preconditioning Strategies
• Physics-constrained neural networks for half-space seismic wave modeling
• A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics
• Data‐Efficient Electromagnetic Surrogate Solver Through Dissipative Relaxation Transfer Learning
• Learning Low Rank Neural Representations of Hyperbolic Wave Dynamics from Data
• Unlocking Out-of-Distribution Generalization in Dynamics through Physics-Guided Augmentation
• A Practitioner's Guide to Kolmogorov-Arnold Networks
• Training Deep Physics-Informed Kolmogorov-Arnold Networks
• Self-induced stochastic resonance: A physics-informed machine learning approach
• Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features
• MNO: Multiscale Neural Operator for 3D Computational Fluid Dynamics
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
• Trace Regularity PINNs: Enforcing H^(1)/(2)(∂ Ω) for Boundary Data
• Causality-guided adaptive sampling method for physics-informed neural networks solving forward problems of partial differential equations
• General Fourier Feature Physics-Informed Extreme Learning Machine (GFF-PIELM) for High-Frequency PDEs
• Deep Neural Networks Inspired by Differential Equations
• AutoBalance: An Automatic Balancing Framework for Training Physics-Informed Neural Networks
• RamPINN: Recovering Raman Spectra From Coherent Anti-Stokes Spectra Using Embedded Physics
• Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
• Physics-Informed Machine Learning in Biomedical Science and Engineering
• Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding
• Modeling Terahertz Propagation via Frequency-Domain Physics-Informed Neural Networks
• Differentiable physics for sound field reconstruction
• A physics-informed neural network approach to the point defect model for electrochemical oxide film growth
• Gated X-TFC: Soft Domain Decomposition for Forward and Inverse Problems in Sharp-Gradient PDEs
• On the joint observability of flow fields and particle properties from Lagrangian trajectories: evidence from neural data assimilation
• Physics Informed Neural Networks for design optimisation of diamond particle detectors for charged particle fast-tracking at high luminosity hadron colliders
• Sobolev acceleration for neural networks
• Learning the solution operator of parametric partial differential equations with physics-informed DeepONets
• Breaking the Discretization Barrier of Continuous Physics Simulation Learning
• R-PINN: Recovery-type a-posteriori estimator enhanced adaptive PINN
• A Neural Network for the Identical Kuramoto Equation: Architectural Considerations and Performance Evaluation
• A Variational Framework for Residual-Based Adaptivity in Neural PDE Solvers and Operator Learning
• Semi-Discrete in Time Method for Time-Dependent Equations by Random Neural Basis
• Multi-Objective Loss Balancing in Physics-Informed Neural Networks for Fluid Flow Applications
• Discovery of Unstable Singularities
• Unified Spatiotemporal Physics-Informed Learning (USPIL): A Framework for Modeling Complex Predator-Prey Dynamics
• Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions
• Physics-Informed Kolmogorov-Arnold Networks for multi-material elasticity problems in electronic packaging
• ReBaNO: Reduced Basis Neural Operator Mitigating Generalization Gaps and Achieving Discretization Invariance
• Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning
• Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need
• A Physics-Informed Neural Networks-Based Model Predictive Control Framework for SIR Epidemics
• HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions
• Neuro-Spectral Architectures for Causal Physics-Informed Networks
• Initialization Schemes for Kolmogorov-Arnold Networks: An Empirical Study
• Quasi-van Hove singularities informed approach improving DOS/pDOS predictions in GNN
• A Hybrid Discontinuous Galerkin Neural Network Method for Solving Hyperbolic Conservation Laws with Temporal Progressive Learning
• Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation
• Fast Convergence Rates for Subsampled Natural Gradient Algorithms on Quadratic Model Problems
• Physics-Constrained Machine Learning for Chemical Engineering
• D3PINNs: A Novel Physics-Informed Neural Network Framework for Staged Solving of Time-Dependent Partial Differential Equations
• Objective Value Change and Shape-Based Accelerated Optimization for the Neural Network Approximation
• Physics-Informed DeepONet Coupled with FEM for Convective Transport in Porous Media with Sharp Gaussian Sources
• Numerical Analysis of Unsupervised Learning Approaches for Parameter Identification in PDEs
• Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next
• Finite-Width Neural Tangent Kernels from Feynman Diagrams
• Noise-adapted Neural Operator for Robust Non-Line-of-Sight Imaging
• Improved Training Strategies for Physics-Informed Neural Networks using Real Experimental Data in Aluminum Spot Welding
• Solved in Unit Domain: JacobiNet for Differentiable Coordinate-Transformed PINNs
• Extended Interface Physics-Informed Neural Networks Method for Moving Interface Problems
• Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces
• Multi-Resolution Training-Enhanced Kolmogorov-Arnold Networks for Multi-Scale PDE Problems
• Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation
• Low-rank adaptive physics-informed HyperDeepONets for solving differential equations
• Physics-informed neural network with pretraining optimization for ocean acoustic field prediction
• LEAD: Exploring Logit Space Evolution for Model Selection
• Physics-Informed Linear Model (PILM): Analytical Representations and Application to Crustal Strain Rate Estimation
• Reversible and irreversible bracket-based dynamics for deep graph neural networks
• PINNeik: Eikonal solution using physics-informed neural networks
• Kernel-Adaptive PI-ELMs for Forward and Inverse Problems in PDEs with Sharp Gradients
• Discontinuity-aware KAN-based physics-informed neural networks
• Simulating Three-dimensional Turbulence with Physics-informed Neural Networks
• Physics-Informed Neural Networks with Hard Nonlinear Equality and Inequality Constraints
• Quasi-Random Physics-informed Neural Networks
• Towards Robust Surrogate Models: Benchmarking Machine Learning Approaches to Expediting Phase Field Simulations of Brittle Fracture
• Noisy PDE Training Requires Bigger PINNs
• Fredholm Neural Networks for inverse problems in elliptic PDEs
• A posteriori analysis of neural network approximations
• Physics-informed neural networks combined with polynomial interpolation to solve nonlinear partial differential equations
• Variable separated physics-informed neural networks based on adaptive weighted loss functions for blood flow model
• Physical Informed Neural Networks for modeling ocean pollutant
• Machine Learning in Acoustics: A Review and Open-Source Repository
• Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs
• Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systems
• A Neural Operator based on Dynamic Mode Decomposition
• Neural Tangent Kernel Analysis to Probe Convergence in Physics-informed Neural Solvers: PIKANs vs. PINNs
• Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks
• Convolution-weighting method for the physics-informed neural network: A Primal-Dual Optimization Perspective
• Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networks
• Robust Physics-Informed Neural Network Approach for Estimating Heterogeneous Elastic Properties from Noisy Displacement Data
• Transformed Diffusion-Wave fPINNs: Enhancing Computing Efficiency for PINNs Solving Time-Fractional Diffusion-Wave Equations
• Structure and asymptotic preserving deep neural surrogates for uncertainty quantification in multiscale kinetic equations
• DeePoly: A High-Order Accuracy Scientific Machine Learning Framework for Function Approximation and Solving PDEs
• VC-PINN: Variable coefficient physics-informed neural network for forward and inverse problems of PDEs with variable coefficient
• Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints
• Physics-informed neural network estimation of active material properties in time-dependent cardiac biomechanical models
• Hybrid deep learning and iterative methods for accelerated solutions of viscous incompressible flow
• Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEs
• Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
• iPINNER: An Iterative Physics-Informed Neural Network with Ensemble Kalman Filter
• Physics-Informed Neural Networks for Sparse Strain-Field Reconstruction in 4D-STEM
• Physics-Informed Neural Networks for Complex Eigenfrequency Identification and Mode Structure Reconstruction of the Ground-State ITG Branch
• FlowMixer: A Constrained Neural Architecture for Interpretable Spatiotemporal Forecasting
• Continuous Representation Methods, Theories, and Applications: An Overview and Perspectives
• Deep Learning for Continuous-Time Stochastic Control with Jumps
• Latent Generative Modeling of Random Fields from Limited Training Data
• Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization
• SPIEDiff: robust learning of long-time macroscopic dynamics from short-time particle simulations with quantified epistemic uncertainty
• Dual-Balancing for Physics-Informed Neural Networks
• Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra
• A Physics-Chemistry-Informed Neural Network (PCINN) for Real-Time Spatial-ALD Coverage Prediction and Reliable Kinetics Inversion
• FMEnets: Flow, Material, and Energy networks for non-ideal plug flow reactor design
• Mask-PINNs: Mitigating Internal Covariate Shift in Physics-Informed Neural Networks
• Fast Stochastic Second-Order Adagrad for Nonconvex Bound-Constrained Optimization
• Stochastic-Dimension Frozen Sampled Neural Network for High-Dimensional Gross-Pitaevskii Equations on Unbounded Domains
• Physics-Informed Neural Network Method and Application to Nuclear Reactor Calculations: A Pilot Study
• Integration Matters for Learning PDEs with Backwards SDEs
• Reduced-order structure-property linkages for stochastic metamaterials
• Learning and Transferring Physical Models through Derivatives
• Multi-level datasets training method in Physics-Informed Neural Networks
• Temporal Neural Operator for Modeling Time-Dependent Physical Phenomena
• Reliable and efficient inverse analysis using physics-informed neural networks with normalized distance functions and adaptive weight tuning
• A solver-in-the-loop framework for end-to-end differentiable coastal hydrodynamics
• Flow-based Extremal Mathematical Structure Discovery
• High-Fidelity Modeling of Stochastic Chemical Dynamics on Complex Manifolds: A Multi-Scale SIREN-PINN Framework for the Curvature-Perturbed Ginzburg-Landau Equation
• Multistage physics informed neural network for solving coupled multiphysics problems in material degradation and fluid dynamics
• An Artificial-Compressibility Physics-Informed Neural Network for the Unsteady Incompressible Navier--Stokes Equations
• A solution for finite journal bearings by using physics-informed neural networks with both soft and hard constrains
• Physics-Informed Neural Network Prediction of Thermophysical Properties for Propyl Butyrate + 1-Alkanol (C6–C10)
• Equilibrium Conserving Neural Operators for Super-Resolution Learning
• How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings
• RL-PINNs: Reinforcement Learning-Driven Adaptive Sampling for Efficient Training of PINNs
• PINNs for Solving Unsteady Maxwell's Equations: Convergence Issues and Comparative Assessment with Compact Schemes
• BO-SA-PINNs: Self-adaptive physics-informed neural networks based on Bayesian optimization for automatically designing PDE solvers
• A residual weighted physics informed neural network for forward and inverse problems of reaction diffusion equations
• Hybrid Temporal Differential Consistency Autoencoder for Efficient and Sustainable Anomaly Detection in Cyber-Physical Systems
• PINNverse: Accurate parameter estimation in differential equations from noisy data with constrained physics-informed neural networks
• Physics-informed neural networks for myocardial perfusion MRI quantification. [europepmc]
• A Feature-Encoded Physics-Informed Parameter Identification Neural Network for Musculoskeletal Systems. [europepmc]
• Physics-Informed Neural Networks for Solving Coupled Stokes-Darcy Equation. [europepmc]
• Investigating molecular transport in the human brain from MRI with physics-informed neural networks. [europepmc]
• Dynamic Weight Strategy of Physics-Informed Neural Networks for the 2D Navier-Stokes Equations. [europepmc]
• A Physics-Guided Neural Operator Learning Approach to Model Biological Tissues From Digital Image Correlation Measurements. [europepmc]
• Physics Informed Neural Networks (PINN) for Low Snr Magnetic Resonance Electrical Properties Tomography (MREPT). [europepmc]
• A Second-Order Network Structure Based on Gradient-Enhanced Physics-Informed Neural Networks for Solving Parabolic Partial Differential Equations. [europepmc]
• Affine transformations accelerate the training of physics-informed neural networks of a one-dimensional consolidation problem. [europepmc]
• h-Analysis and data-parallel physics-informed neural networks. [europepmc]
• Current status in spatiotemporal analysis of contrast-based perfusion MRI. [europepmc]
• Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks. [europepmc]
• Physics-informed UNets for discovering hidden elasticity in heterogeneous materials. [europepmc]
• Continuous and discontinuous compressible flows in a converging-diverging channel solved by physics-informed neural networks without exogenous data. [europepmc]
• SELFNet: Denoising Shear Wave Elastography Using Spatial-temporal Fourier Feature Networks. [europepmc]
• Modeling transient mixed flows in sewer systems with data fusion via physics-informed machine learning. [europepmc]
• Accelerated optimization in deep learning with a proportional-integral-derivative controller. [europepmc]
• Adapting physics-informed neural networks to improve ODE optimization in mosquito population dynamics. [europepmc]
• PINNing cerebral blood flow: analysis of perfusion MRI in infants using physics-informed neural networks. [europepmc]
• Physics informed neural networks for fluid flow analysis with repetitive parameter initialization. [europepmc]
• Physics-informed neural networks for physiological signal processing and modeling: a narrative review. [europepmc]
• One-shot learning for solution operators of partial differential equations. [europepmc]
• Temporal neural operator for modeling time-dependent physical phenomena. [europepmc]
• Gradient-Driven Physics Informed Neural Networks for Conduction Heat Transfer and Incompressible Laminar Flow. [europepmc]
• Physics-Informed Emulation of Systemic Circulation for Fast Parameter Estimation and Uncertainty Quantification. [europepmc]