Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs

2021/01/20 by Jie Bu, Anuj Karpatne, Bu, Jie +1 · 8 citations
Computer Science · Engineering · Physics and Astronomy · #Advanced Numerical Analysis Techniques #Computational Engineering #FOS: Computer and information sciences #Finance #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #and Science (cs.CE)

paper · pdf · doi:10.48550/arxiv.2101.08366

openalex publication_date 2021/01/20 · openalex created_date 2021/09/13 · openalex updated_date 2026/07/28

Abstract

We propose quadratic residual networks (QRes) as a new type of parameter-efficient neural network architecture, by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. With sufficiently high functional capacity (or expressive power), we show that it is especially powerful for solving forward and inverse physics problems involving partial differential equations (PDEs). Using tools from algebraic geometry, we theoretically demonstrate that, in contrast to plain neural networks, QRes shows better parameter efficiency in terms of network width and depth thanks to higher non-linearity in every neuron. Finally, we empirically show that QRes shows faster convergence speed in terms of number of training epochs especially in learning complex patterns.

Cited by

• Perturbative-NeuSA: A Structured Spectral Framework for Time-Dependent PDEs
• Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
• The Alexander-Hirschowitz theorem for neurovarieties
• Addressing the gravitational collapse of a massless scalar field with Physics-Informed Neural Networks
• General transformation neural networks: A class of parametrized functions for high-dimensional function approximation
• Neuro-Spectral Architectures for Causal Physics-Informed Networks
• PIANO: Physics Informed Autoregressive Network
• Revisiting Heat Flux Analysis of Tungsten Monoblock Divertor on EAST using Physics-Informed Neural Network

Related

• Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning
• Connections between Numerical Algorithms for PDEs and Neural Networks
• Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations
• Applications and Manipulations of Physics-Informed Neural Networks in Solving Differential Equations
• Momentum Residual Neural Networks