Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery

2025/05/29 by Ning Liu, Yue Yu, Liu, Ning +1 · 1 citation
Computer Science · Materials Science · Physics and Astronomy · #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Numerical Analysis (math.NA)

paper · pdf · doi:10.48550/arxiv.2505.23106

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Attention mechanisms have emerged as transformative tools in core AI domains such as natural language processing and computer vision. Yet, their largely untapped potential for modeling intricate physical systems presents a compelling frontier. Learning such systems often entails discovering operators that map between functional spaces using limited instances of function pairs -- a task commonly framed as a severely ill-posed inverse PDE problem. In this work, we introduce Neural Interpretable PDEs (NIPS), a novel neural operator architecture that builds upon and enhances Nonlocal Attention Operators (NAO) in both predictive accuracy and computational efficiency. NIPS employs a linear attention mechanism to enable scalable learning and integrates a learnable kernel network that acts as a channel-independent convolution in Fourier space. As a consequence, NIPS eliminates the need to explicitly compute and store large pairwise interactions, effectively amortizing the cost of handling spatial interactions into the Fourier transform. Empirical evaluations demonstrate that NIPS consistently surpasses NAO and other baselines across diverse benchmarks, heralding a substantial leap in scalable, interpretable, and efficient physics learning. Our code and data accompanying this paper are available at https://github.com/fishmoon1234/Nonlocal-Attention-Operator.

Citations

• Transformer learns the cross-task prior and regularization for in-context learning
• Monotone Peridynamic Neural Operator for Nonlinear Material Modeling with Conditionally Unique Solutions
• Physics-Informed Deep Inverse Operator Networks for Solving PDE Inverse Problems
• Pretrained transformer efficiently learns low-dimensional target functions in-context
• Disentangled Representation Learning for Parametric Partial Differential Equations
• Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery
• Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
• Asymptotic theory of in-context learning by linear attention
• Deep Neural Operator Enabled Digital Twin Modeling for Additive Manufacturing
• Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation
• MODNO: Multi Operator Learning With Distributed Neural Operators
• Polynormer: Polynomial-Expressive Graph Transformer in Linear Time
• Linear Transformers are Versatile In-Context Learners
• PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations
• PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws
• Peridynamic Neural Operators: A Data-Driven Nonlocal Constitutive Model for Complex Material Responses
• Let data talk: data-regularized operator learning theory for inverse problems
• Domain Agnostic Fourier Neural Operators
• GNOT: A General Neural Operator Transformer for Operator Learning
• Neural Inverse Operators for Solving PDE Inverse Problems
• INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation
• Reinforced Inverse Scattering
• Transformer for Partial Differential Equations' Operator Learning
• IAE-Net: Integral Autoencoders for Discretization-Invariant Learning
• Coupling Deep Learning with Full Waveform Inversion
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• Physics-Informed Neural Operator for Learning Partial Differential Equations
• Choose a Transformer: Fourier or Galerkin
• Physics-informed machine learning
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Physics-informed neural networks (PINNs) for fluid mechanics: a review
• Rethinking Attention with Performers
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data
• Neural Operator: Graph Kernel Network for Partial Differential Equations
• Deep synthesis regularization of inverse problems
• Solving Traveltime Tomography with Deep Learning
• Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
• Ab initio solution of the many-electron Schrödinger equation with deep neural networks
• Machine learning and the physical sciences
• Regularization by Architecture: A Deep Prior Approach for Inverse Problems
• Inverse problems for the stationary transport equation in the diffusion scaling
• Attention Is All You Need
• PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels
• Mitigating spectral bias for the multiscale operator learning

Cited by

• Learning Causal Graphs at Scale: A Foundation Model Approach
• Spectral-inspired Operator Learning with Limited Data and Unknown Physics

Related

• Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery
• LFR-PINO: A Layered Fourier Reduced Physics-Informed Neural Operator for Parametric PDEs
• Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
• Latent Neural Operator for Solving Forward and Inverse PDE Problems
• Integrating Locality-Aware Attention with Transformers for General Geometry PDEs