A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep\n Feed-Forward Neural Networks

2024/12/12 by Kyle R. Chickering, Chickering, Kyle R.
Computer Science · #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2412.09752

Abstract

The use of neural networks for solving differential equations is practically\ndifficult due to the exponentially increasing runtime of autodifferentiation\nwhen computing high-order derivatives. We propose n-TangentProp, the natural\nextension of the TangentProp formalism citesimard1991tangent to arbitrarily\nmany derivatives. n-TangentProp computes the exact derivative dⁿ/dxⁿ f(x)\nin quasilinear, instead of exponential time, for a densely connected,\nfeed-forward neural network f with a smooth, parameter-free activation\nfunction. We validate our algorithm empirically across a range of depths,\nwidths, and number of derivatives. We demonstrate that our method is\nparticularly beneficial in the context of physics-informed neural networks\nwhere ntp allows for significantly faster training times than previous methods\nand has favorable scaling with respect to both model size and loss-function\ncomplexity as measured by the number of required derivatives. The code for this\npaper can be found at https://github.com/kyrochi/n\_tangentprop.\n

Related

• Backpropagation generalized for output derivatives
• Derivative-enhanced Deep Operator Network
• DOF: Accelerating High-order Differential Operators with Forward Propagation
• Neural Networks with Cheap Differential Operators
• Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learning