A Residual Guided strategy with Generative Adversarial Networks in training Physics-Informed Transformer Networks

2025/07/15 by Ziyang Zhang, Zhang, Ziyang, Feifan Zhang +7
Physics and Astronomy · Materials Science · #Model Reduction and Neural Networks #Machine Learning in Materials Science #Quantum many-body systems

paper · pdf · doi:10.48550/arxiv.2508.00855

Abstract

Nonlinear partial differential equations (PDEs) are pivotal in modeling complex physical systems, yet traditional Physics-Informed Neural Networks (PINNs) often struggle with unresolved residuals in critical spatiotemporal regions and violations of temporal causality. To address these limitations, we propose a novel Residual Guided Training strategy for Physics-Informed Transformer via Generative Adversarial Networks (GAN). Our framework integrates a decoder-only Transformer to inherently capture temporal correlations through autoregressive processing, coupled with a residual-aware GAN that dynamically identifies and prioritizes high-residual regions. By introducing a causal penalty term and an adaptive sampling mechanism, the method enforces temporal causality while refining accuracy in problematic domains. Extensive numerical experiments on the Allen-Cahn, Klein-Gordon, and Navier-Stokes equations demonstrate significant improvements, achieving relative MSE reductions of up to three orders of magnitude compared to baseline methods. This work bridges the gap between deep learning and physics-driven modeling, offering a robust solution for multiscale and time-dependent PDE systems.

Citations

• Respecting causality for training physics-informed neural networks
• Self-Scalable Tanh (Stan): Multi-Scale Solutions for Physics-Informed Neural Networks
• PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs
• Adversarial Adaptive Sampling: Unify PINN and Optimal Transport for the Approximation of PDEs
• Failure-informed adaptive sampling for PINNs
• DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations
• DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations
• Active Learning Based Sampling for High-Dimensional Nonlinear Partial Differential Equations
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics
• Physics-informed machine learning
• Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
• Parallel Physics-Informed Neural Networks via Domain Decomposition
• Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Fourier Neural Operator for Parametric Partial Differential Equations
• Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks
• Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems
• Physics informed deep learning for computational elastodynamics without labeled data
• Extended Physics-Informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition Based Deep Learning Framework for Nonlinear Partial Differential Equations
• Neural Operator: Graph Kernel Network for Partial Differential Equations
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• DeepXDE: A Deep Learning Library for Solving Differential Equations
• Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks
• Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Attention Is All You Need
• Adam: A Method for Stochastic Optimization
• Soliton solution of some nonlinear partial differential equations and its applications in fluid mechanics
• On the limited memory BFGS method for large scale optimization

Related

• Causality-guided adaptive sampling method for physics-informed neural networks
• PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks
• PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks
• Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning
• A Novel Adaptive Causal Sampling Method for Physics-Informed Neural Networks