Hybrid Least Squares/Gradient Descent Methods for DeepONets

2025/08/21 by Jun Woo Choi, Choi, Jun, Chang-Ock Lee +3
Computer Science · Physics and Astronomy · #Stochastic Gradient Optimization Techniques #Model Reduction and Neural Networks #Advanced Neural Network Applications

paper · pdf · doi:10.48550/arxiv.2508.15394

Abstract

We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems \unicodex2014 one for the branch network and one for the trunk network \unicodex2014 and solves them separately. This method is generalized to a broader type of L² loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.

Citations

• ELM-DeepONets: Backpropagation-Free Training of Deep Operator Networks via Extreme Learning Machines
• A Survey on Universal Approximation Theorems
• Partitioned neural network approximation for partial differential equations enhanced with Lagrange multipliers and localized loss functions
• MIONet: Learning multiple-input operators via tensor product
• MIONet: Learning Multiple-Input Operators via Tensor Product
• Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets
• Learning the solution operator of parametric partial differential equations with physics-informed DeepONets
• Fourier Neural Operator for Parametric Partial Differential Equations
• Neural Operator: Graph Kernel Network for Partial Differential Equations
• Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint
• Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• ResNet with one-neuron hidden layers is a Universal Approximator
• Universality of Deep Convolutional Neural Networks
• Universality of deep convolutional neural networks
• The Expressive Power of Neural Networks: A View from the Width
• Accelerating Deep Learning with Shrinkage and Recall
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Automatic differentiation in machine learning: a survey
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Adam: A Method for Stochastic Optimization
• Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems
• On the limited memory BFGS method for large scale optimization
• Multilayer feedforward networks are universal approximators
• The Commutation Matrix: Some Properties and Applications

Related

• Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables
• Learning Gradient Descent: Better Generalization and Longer Horizons
• A Solver + Gradient Descent Training Algorithm for Deep Neural Networks
• ELM-DeepONets: Backpropagation-Free Training of Deep Operator Networks via Extreme Learning Machines
• Gradient Descent based Optimization Algorithms for Deep Learning Models Training