Automatic differentiation in machine learning: a survey
2015/02/20 by Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul +1 · 3 voices · 123 citations
#cs.SC #cs.LG #stat.ML
paper · pdf
Abstract
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.
Cited by
- Parametric Neural r-Adaptivity for Isogeometric Analysis via Residual Minimization
- PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs
- Efficient optimisation of multi-parameter quantum control protocols for strongly-coupled systems
- Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems
- A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing
- Backpropagating Pauli Propagation
- Higher-Order Hit-&-Run Samplers for Linearly Constrained Densities
- Machine learning enabled multiscale model for nanoparticle margination and physiology based pharmacokinetics
- A Common Interface for Automatic Differentiation
- Training Large Neural Networks With Low-Dimensional Error Feedback
- Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation
- SURE Guided Posterior Sampling: Trajectory Correction for Diffusion-Based Inverse Problems
- Breaking the Memory Wall: Exact Analytical Differentiation via Tiled Operator-Space Evolution
- PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations
- Differentiable Inverse Modeling with Physics-Constrained Latent Diffusion for Heterogeneous Subsurface Parameter Fields
- A PINN approach for the online identification and control of unknown PDEs
- QuTiP 5: The Quantum Toolbox in Python
- Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
- jaxdae: A JAX-native Differentiable Solver for Differential-Algebraic Equations in Coupled Multi-physics
- Extracting Exact Lie Derivatives Without Backpropagation: A Dual Compiler for Neural Control Barrier Functions
- Optimize discrete loss with finite-difference physics constraint and time-stepping for PDE solving
- Non-stationary Spatial Modeling Using Fractional SPDEs
- Exploring the Effect of Basis Rotation on NQS Performance
- Learning Model Parameter Dynamics in a Combination Therapy for Bladder Cancer from Sparse Biological Data
- Stability and complexity of global iterative solvers for the Kadanoff-Baym equations
- Structural Optimization in Tensor LEED Using a Parameter Tree and R-Factor Gradients
- Wavelet-Accelerated Physics-Informed Quantum Neural Network for Multiscale Partial Differential Equations
- Rheological Parameter Identification in Granular Materials Using Physics-Informed Neural Networks
- DAE-HardNet: A Physics Constrained Neural Network Enforcing Differential-Algebraic Hard Constraints
- MDcraft -- a modern molecular dynamics simulation package with machine learning potentials support
- Active learning with physics-informed neural networks for optimal sensor placement in deep tunneling through transversely isotropic elastic rocks
- Physics Enhanced Deep Surrogates for the Phonon Boltzmann Transport Equation
- Solving Heterogeneous Agent Models with Physics-informed Neural Networks
- Eigenvalue-accelerated LDOS optimization of high-Q optical resonances
- Addressing the gravitational collapse of a massless scalar field with Physics-Informed Neural Networks
- Gradient-descent methods for quantum detector tomography
- Cost-Driven Synthesis of Sound Abstract Interpreters
- Real-Time Physics-Aware Battery Health Monitoring from Partial Charging Profiles via Physics-Informed Neural Networks
- Batch Matrix-form Equations and Implementation of Multilayer Perceptrons
- Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
- Learning to Fast Unrank in Collaborative Filtering Recommendation
- Events Meet Phase-Shifting Digital Holography: Practical Acquisition, Theory, and Algorithms
- Zero-Shot Function Encoder-Based Differentiable Predictive Control
- Nonparametric Modeling of Continuous-Time Markov Chains
- Dynamic Reconstruction of Ultrasound-Derived Flow Fields With Physics-Informed Neural Fields
- Physics-Informed Neural Network-based Reliability Analysis of Buried Pipelines
- Sensitivity Analysis for Climate Science with Generative Flow Models
- Learning Soil Physics from Partial Knowledge and Data: Partitioning Capillary and Adsorbed Soil Water
- Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization
- Equation Discovery, Parametric Simulation, and Optimization Using the Physics-Informed Neural Network (PINN) Method for the Heat Conduction Problem
- Spatiotemporal control of laser intensity using differentiable programming
- Physics-Informed Broad Learning System: An Efficient Backpropagation-Free Framework for Solving Partial Differential Equations
- GradInf: Gradient Estimation as Probabilistic Inference
- Training Deep Physics-Informed Kolmogorov-Arnold Networks
- Aiming for Proxima Centauri b: Gravitational effects on relativistic spacecraft trajectories
- Expectation-propagation for Bayesian empirical likelihood inference
- Information Gradient for Nonlinear Gaussian Channel with Applications to Task-Oriented Communication
- Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
- A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
- On Biologically Plausible Learning in Continuous Time
- A Frequentist Statistical Introduction to Variational Inference, Autoencoders, and Diffusion Models
- Deep Neural Network extraction of Unpolarized Transverse Momentum Distributions
- Trajectory Optimization for Minimum Threat Exposure using Physics-Informed Neural Networks
- CBINNS: Cancer Biology-Informed Neural Network for Unknown Parameter Estimation and Missing Physics Identification
- On the use of the Derivative Approximation for Likelihoods for Gravitational Wave Inference
- State estimation in homogeneous isotropic turbulence using super-resolution with a 4DVar training algorithm
- Boundary-Informed Method of Lines for Physics Informed Neural Networks
- Jet Functors and Weil Algebras in Automatic Differentiation: A Geometric Analysis
- Numerical Approximation of Electrohydrodynamics Model: A Comparative Study of PINNs and FEM
- Modeling Adoptive Cell Therapy in Bladder Cancer from Sparse Biological Data using PINNs
- SCOPED: Score-Curvature Out-of-distribution Proximity Evaluator for Diffusion
- Neural Network Surrogates for Free Energy Computation of Complex Chemical Systems
- Physics-Informed Machine Learning in Biomedical Science and Engineering
- Differentiable physics for sound field reconstruction
- Feature Identification via the Empirical NTK
- Bundle Network: a Machine Learning-Based Bundle Method
- Coupling Physics Informed Neural Networks with External Solvers
- Enabling Physical AI through Biological Principles
- Causally-Enhanced Reinforcement Policy Optimization
- Zeroth-order parallel sampling
- Learning the Helmholtz equation operator with DeepONet for non-parametric 2D geometries
- Generative modeling with Gaussian Boson Sampling: classically trainable Bosonic Born Machines
- An Improved Yau-Yau Algorithm for High Dimensional Nonlinear Filtering Problems
- TOFLUX: A Differentiable Topology Optimization Framework for Multiphysics Fluidic Problems
- Compose Yourself: Average-Velocity Flow Matching for One-Step Speech Enhancement
- Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
- PBPK-iPINNs: Inverse Physics-Informed Neural Networks for Physiologically Based Pharmacokinetic Brain Models
- Curvature Learning for Generalization of Hyperbolic Neural Networks
- Neuromorphic Intelligence
- Hetero-EUCLID: Interpretable model discovery for heterogeneous hyperelastic materials using stress-unsupervised learning
- Learning Singularity-Encoded Green's Functions with Application to Iterative Methods
- Gradients, parallelism, and variance of quantum estimates
- A Differentiable Surrogate Model for the Generation of Radio Pulses from In-Ice Neutrino Interactions
- Variational Neural Networks for Observable Thermodynamics (V-NOTS)
- Physics-informed waveform inversion using pretrained wavefield neural operators
- Algorithmic differentiation for plane-wave DFT: materials design, error control and learning model parameters
- IP-Basis PINNs: Efficient Multi-Query Inverse Parameter Estimation
- A deep learning-driven iterative scheme for high-dimensional HJB equations in portfolio selection with exogenous and endogenous costs
- DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing
- A constrained optimization approach to nonlinear system identification through simulation error minimization
- Data-driven modeling for flow reconstruction from sparse temperature measurements
- Learning by training: emergent return-point memory from cyclically tuning disordered sphere packings
- Assessing the Advantages and Limitations of Quantum Neural Networks in Regression Tasks
- Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation
- Fast approximate Bayesian inference of HIV indicators using PCA adaptive Gauss-Hermite quadrature
- Physics-Informed DeepONet Coupled with FEM for Convective Transport in Porous Media with Sharp Gaussian Sources
- Differentiable multiphase flow model for physics-informed machine learning in reservoir pressure management
- Numerical Analysis of Unsupervised Learning Approaches for Parameter Identification in PDEs
- Hybrid Least Squares/Gradient Descent Methods for DeepONets
- Strategies for training point distributions in physics-informed neural networks
- HFBTHO-AD: Differentiation of a nuclear energy density functional code
- High-dimensional maximum-entropy phase space tomography
- Temporal Anchoring in Deepening Embedding Spaces: Event-Indexed Projections, Drift, Convergence, and an Internal Computational Architecture
- Extended parameter shift rules with minimal derivative variance for parameterized quantum circuits
- Fast, Convex and Conditioned Network for Multi-Fidelity Vectors and Stiff Univariate Differential Equations
- Hybrid Physics-Machine Learning Models for Quantitative Electron Diffraction Refinements
- MPS-JuliQAOA: User-friendly, Scalable MPS-based Simulation for Quantum Optimization
- Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
- The Vanishing Gradient Problem for Stiff Neural Differential Equations
- Fine-tuning physics-informed neural networks for cavity flows using coordinate transformation
- Graph Lineages and Skeletal Graph Products
- Comparison of deep learning software [wikipedia]
- Physics-informed neural networks [wikipedia]
Discussions
Related