Automatic differentiation in machine learning: a survey
2015/02/20 by Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul +1 · 3 voices · 209 citations
Computer Science · Mathematics · #acm:65D25 #acm:68T05 #acm:68W30 #cs.LG #cs.SC #msc:65D25 #msc:68T05 #msc:68W30 #stat.ML
paper · pdf
published as Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Research, 18(153):1--43, 2018 · 43 pages, 5 figures
arxiv created 2018/02/05 · arxiv updated 2018/07/18
Abstract
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.
Cited by
- Parametric Neural r-Adaptivity for Isogeometric Analysis via Residual Minimization
- PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs
- Efficient optimisation of multi-parameter quantum control protocols for strongly-coupled systems
- Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems
- A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing
- Backpropagating Pauli Propagation
- Higher-Order Hit-&-Run Samplers for Linearly Constrained Densities
- Machine learning enabled multiscale model for nanoparticle margination and physiology based pharmacokinetics
- A Common Interface for Automatic Differentiation
- Training Large Neural Networks With Low-Dimensional Error Feedback
- Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation
- SURE Guided Posterior Sampling: Trajectory Correction for Diffusion-Based Inverse Problems
- Breaking the Memory Wall: Exact Analytical Differentiation via Tiled Operator-Space Evolution
- PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations
- Differentiable Inverse Modeling with Physics-Constrained Latent Diffusion for Heterogeneous Subsurface Parameter Fields
- A PINN approach for the online identification and control of unknown PDEs
- QuTiP 5: The Quantum Toolbox in Python
- Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
- jaxdae: A JAX-native Differentiable Solver for Differential-Algebraic Equations in Coupled Multi-physics
- Extracting Exact Lie Derivatives Without Backpropagation: A Dual Compiler for Neural Control Barrier Functions
- Optimize discrete loss with finite-difference physics constraint and time-stepping for PDE solving
- Non-stationary Spatial Modeling Using Fractional SPDEs
- Exploring the Effect of Basis Rotation on NQS Performance
- Learning Model Parameter Dynamics in a Combination Therapy for Bladder Cancer from Sparse Biological Data
- Stability and complexity of global iterative solvers for the Kadanoff-Baym equations
- Structural Optimization in Tensor LEED Using a Parameter Tree and R-Factor Gradients
- Wavelet-Accelerated Physics-Informed Quantum Neural Network for Multiscale Partial Differential Equations
- Rheological Parameter Identification in Granular Materials Using Physics-Informed Neural Networks
- DAE-HardNet: A Physics Constrained Neural Network Enforcing Differential-Algebraic Hard Constraints
- MDcraft -- a modern molecular dynamics simulation package with machine learning potentials support
- Active learning with physics-informed neural networks for optimal sensor placement in deep tunneling through transversely isotropic elastic rocks
- Physics Enhanced Deep Surrogates for the Phonon Boltzmann Transport Equation
- Solving Heterogeneous Agent Models with Physics-informed Neural Networks
- Eigenvalue-accelerated LDOS optimization of high-Q optical resonances
- Addressing the gravitational collapse of a massless scalar field with Physics-Informed Neural Networks
- Gradient-descent methods for scalable quantum detector tomography
- Cost-Driven Synthesis of Sound Abstract Interpreters
- Real-Time Physics-Aware Battery Health Monitoring from Partial Charging Profiles via Physics-Informed Neural Networks
- Batch Matrix-form Equations and Implementation of Multilayer Perceptrons
- Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
- Learning to Fast Unrank in Collaborative Filtering Recommendation
- Events Meet Phase-Shifting Digital Holography: Practical Acquisition, Theory, and Algorithms
- Zero-Shot Function Encoder-Based Differentiable Predictive Control
- Nonparametric Modeling of Continuous-Time Markov Chains
- Dynamic Reconstruction of Ultrasound-Derived Flow Fields With Physics-Informed Neural Fields
- Physics-Informed Neural Network-based Reliability Analysis of Buried Pipelines
- Sensitivity Analysis for Climate Science with Generative Flow Models
- Learning Soil Physics from Partial Knowledge and Data: Partitioning Capillary and Adsorbed Soil Water
- Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization
- Equation Discovery, Parametric Simulation, and Optimization Using the Physics-Informed Neural Network (PINN) Method for the Heat Conduction Problem
- Spatiotemporal control of laser intensity using differentiable programming
- Physics-Informed Broad Learning System: An Efficient Backpropagation-Free Framework for Solving Partial Differential Equations
- GradInf: Gradient Estimation as Probabilistic Inference
- Training Deep Physics-Informed Kolmogorov-Arnold Networks
- Aiming for Proxima Centauri b: Gravitational effects on relativistic spacecraft trajectories
- Expectation-Propagation for Bayesian Empirical Likelihood Inference
- Information Gradient for Nonlinear Gaussian Channel with Applications to Task-Oriented Communication
- Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
- A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
- Does Feedback Alignment Work at Biological Timescales?
- A Frequentist Statistical Introduction to Variational Inference, Autoencoders, and Diffusion Models
- Deep Neural Network extraction of Unpolarized Transverse Momentum Distributions
- Trajectory Optimization for Minimum Threat Exposure using Physics-Informed Neural Networks
- CBINNS: Cancer Biology-Informed Neural Network for Unknown Parameter Estimation and Missing Physics Identification
- On the use of the Derivative Approximation for Likelihoods for Gravitational Wave Inference
- State estimation in homogeneous isotropic turbulence using super-resolution with a 4DVar training algorithm
- Boundary-Informed Method of Lines for Physics Informed Neural Networks
- Jet Functors and Weil Algebras in Automatic Differentiation: A Geometric Analysis
- Numerical Approximation of Electrohydrodynamics Model: A Comparative Study of PINNs and FEM
- Modeling Adoptive Cell Therapy in Bladder Cancer from Sparse Biological Data using PINNs
- SCOPED: Score-Curvature Out-of-distribution Proximity Evaluator for Diffusion
- Neural Network Surrogates for Free Energy Computation of Complex Chemical Systems
- Physics-Informed Machine Learning in Biomedical Science and Engineering
- Differentiable physics for sound field reconstruction
- Feature Identification via the Empirical NTK
- Bundle Network: a Machine Learning-Based Bundle Method
- Coupling Physics Informed Neural Networks with External Solvers
- Enabling Physical AI through Biological Principles
- Causally-Enhanced Reinforcement Policy Optimization
- Zeroth-order parallel sampling
- Learning the Helmholtz equation operator with DeepONet for non-parametric 2D geometries
- Generative modeling with Gaussian Boson Sampling: classically trainable Bosonic Born Machines
- An Improved Yau-Yau Algorithm for High Dimensional Nonlinear Filtering Problems
- TOFLUX: A Differentiable Topology Optimization Framework for Multiphysics Fluidic Problems
- Compose Yourself: Average-Velocity Flow Matching for One-Step Speech Enhancement
- Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
- PBPK-iPINNs: Inverse Physics-Informed Neural Networks for Physiologically Based Pharmacokinetic Brain Models
- Curvature Learning for Generalization of Hyperbolic Neural Networks
- Neuromorphic Intelligence
- Hetero-EUCLID: Interpretable model discovery for heterogeneous hyperelastic materials using stress-unsupervised learning
- Learning Singularity-Encoded Green's Functions with Application to Iterative Methods
- Gradients, parallelism, and variance of quantum estimates
- A Differentiable Surrogate Model for the Generation of Radio Pulses from In-Ice Neutrino Interactions
- Variational Neural Networks for Observable Thermodynamics (V-NOTS)
- Physics-informed waveform inversion using pretrained wavefield neural operators
- Algorithmic differentiation for plane-wave DFT: materials design, error control and learning model parameters
- IP-Basis PINNs: Efficient Multi-Query Inverse Parameter Estimation
- A deep learning-driven iterative scheme for high-dimensional HJB equations in portfolio selection with exogenous and endogenous costs
- DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing
- A constrained optimization approach to nonlinear system identification through simulation error minimization
- Data-driven modeling for flow reconstruction from sparse temperature measurements
- Learning by training: emergent return-point memory from cyclically tuning disordered sphere packings
- Assessing the Advantages and Limitations of Quantum Neural Networks in Regression Tasks
- Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation
- Fast approximate Bayesian inference of HIV indicators using PCA adaptive Gauss-Hermite quadrature
- Physics-Informed DeepONet Coupled with FEM for Convective Transport in Porous Media with Sharp Gaussian Sources
- Differentiable multiphase flow model for physics-informed machine learning in reservoir pressure management
- Numerical Analysis of Unsupervised Learning Approaches for Parameter Identification in PDEs
- Hybrid Least Squares/Gradient Descent Methods for DeepONets
- Strategies for training point distributions in physics-informed neural networks
- HFBTHO-AD: Differentiation of a nuclear energy density functional code
- High-dimensional maximum-entropy phase space tomography
- Black-box optimization in immunology and beyond: A practical guide to algorithms and future directions
- Temporal Anchoring in Deepening Embedding Spaces: Event-Indexed Projections, Drift, Convergence, and an Internal Computational Architecture
- Extended parameter shift rules with minimal derivative variance for parameterized quantum circuits
- Fast, Convex and Conditioned Network for Multi-Fidelity Vectors and Stiff Univariate Differential Equations
- Hybrid Physics-Machine Learning Models for Quantitative Electron Diffraction Refinements
- MPS-JuliQAOA: User-friendly, Scalable MPS-based Simulation for Quantum Optimization
- Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
- The Vanishing Gradient Problem for Stiff Neural Differential Equations
- Fine-tuning physics-informed neural networks for cavity flows using coordinate transformation
- Graph Lineages and Skeletal Graph Products
- PVD-ONet: A Multi-scale Neural Operator Method for Singularly Perturbed Boundary Layer Problems
- PyBird-JAX: Accelerated inference in large-scale structure with model-independent emulation of one-loop galaxy power spectra
- Neural Preconditioning via Krylov Subspace Geometry
- Interpretable Neural Network Quantum States for Solving the Steady States of the Nonlinear Schrödinger Equation
- Physics-Informed Regression: Parameter Estimation in Parameter-Linear Nonlinear Dynamic Models
- DONUT: Physics-aware Machine Learning for Real-time X-ray Nanodiffraction Analysis
- Learning Acceleration Algorithms for Fast Parametric Convex Optimization with Certified Robustness
- Performance Portable Gradient Computations Using Source Transformation
- Quantum Machine Learning in Multi-Qubit Phase-Space Part I: Foundations
- TMMax: High-performance modeling of multilayer thin-film structures using transfer matrix method with JAX
- A user's guide to PINNs in geometric analysis: lessons from the asymptotic Plateau problem
- A Generalizable Physics-Enhanced State Space Model for Long-Term Dynamics Forecasting in Complex Environments
- WellPINN: Accurate Well Representation for Transient Fluid Pressure Diffusion in Subsurface Reservoirs with Physics-Informed Neural Networks
- Optimizing External Sources for Controlled Burning Plasma in Tokamaks with Neural Ordinary Differential Equations
- Differentiable simulations for particle tracking in accelerators: analysis, benchmarking and optimization
- Preparing spin-squeezed states in Rydberg atom arrays via quantum optimal control
- DFSC: Error-Controlled Differentiable Mittag-Leffler Propagation for Fractional Scientific Machine Learning
- JAX-MPM: A Learning-Augmented Differentiable Meshfree Framework for GPU-Accelerated Lagrangian Simulation and Geophysical Inverse Modeling
- Three-dimensional crustal deformation analysis using physics-informed deep learning
- Generative flow-based warm start of the variational quantum eigensolver
- Unified reconstruction of the Lyman-alpha power spectrum with Hamiltonian Monte Carlo
- Benchmark of Approximate Quantum Chemical and Machine Learning Potentials for Biochemical Proton Transfer Reactions
- Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
- ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts
- Memory Savings at What Cost? A Study of Alternatives to Backpropagation
- Evaluating PDE discovery methods for multiscale modeling of biological signals
- Real-Time Execution of Action Chunking Flow Policies
- Numerical simulation of transient heat conduction with moving heat source using Physics Informed Neural Networks
- Inferring viscoplastic models from velocity fields: a physics-informed neural network approach
- Optimal Navigation in Microfluidics via the Optimization of a Discrete Loss
- Time-dependent density estimation using binary classifiers
- Fully Independent Response in Disordered Solids
- Simulation of parametrized cardiac electrophysiology in three dimensions using physics-informed neural networks
- Beyond diagonal approximations: improved covariance modeling for pulsar timing array data analysis
- A modified Newmark/Newton-Raphson method with automatic differentiation for general nonlinear dynamics analysis
- PINNs Algorithmic Framework for Simulation of Nonlinear Burgers' Type Models
- An approximate Riemann solver approach in Physics-Informed Neural Networks for hyperbolic conservation laws
- A detailed and comprehensive account of fractional Physics-Informed Neural Networks: From implementation to efficiency
- Going beyond density functional theory accuracy: Leveraging experimental data to refine pre-trained machine learning interatomic potentials
- Over-PINNs: Enhancing Physics-Informed Neural Networks via Higher-Order Partial Derivative Overdetermination of PDEs
- Learning long range dependencies through time reversal symmetry breaking
- Physics-Informed Neural Network Approach to Quark-Antiquark Color Flux Tube
- Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution
- Higher-Order Automatic Differentiation Using Symbolic Differential Algebra: Bridging the Gap between Algorithmic and Symbolic Differentiation
- Reconstruction and Prediction of Volterra Integral Equations Driven by Gaussian Noise
- Antenna Q-Factor Topology Optimization with Auxiliary Edge Resistivities
- Learning Geometry and Topology via Multi-Chart Flows
- Learning Interpretable Differentiable Logic Networks for Tabular Regression
- EvoGrad: Metaheuristics in a Differentiable Wonderland
- A decomposition-based robust training of physics-informed neural networks for nearly incompressible linear elasticity
- Broad Spectrum Structure Discovery in Large-Scale Higher-Order Networks
- An introduction to Neural Networks for Physicists
- FESOM2-JAX v1.0: a differentiable shadow of the ocean-sea-ice model FESOM2, cast onto GPUs
- Engineering application of physics-informed neural networks for Saint-Venant torsion
- FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks
- Mollifier Layers: Enabling Efficient High-Order Derivatives in Inverse PDE Learning
- From Preimage Search To Source-Grounded Feature Inversion
- Nova: An End-to-End MLIR Compiler for Deep Learning
- Convergence Properties of PINNs for the Navier-Stokes-Cahn-Hilliard System
- Automatically Differentiable Model Updating (ADiMU): conventional, hybrid, and neural network material model discovery including history-dependency
- Fast and Reliable Gradients for Deformables Across Frictional Contact Regimes
- A Practitioner's Guide to Automatic Kernel Search for Gaussian Processes in Battery Applications
- Learning and Transferring Physical Models through Derivatives
- Neural Networks Enabled Discovery On the Higher-Order Nonlinear Partial Differential Equation of Traffic Dynamics
- Automatic Differentiation in Mixture Models
- Multi-level datasets training method in Physics-Informed Neural Networks
- Local Minima in Quadratic-Penalty Relaxations of Binary Linear Programs
- Fast and General Automatic Differentiation for Finite-State Methods
- End-to-end differentiable network traffic simulation with dynamic route choice
- Smooth Integer Encoding via Integral Balance
- Coherence-based Approximate Derivatives via Web of Affine Spaces Optimization
- A solver-in-the-loop framework for end-to-end differentiable coastal hydrodynamics
- Faster parallel MCMC: Metropolis adjustment is best served warm
- Learning to Optimize by Differentiable Programming
- FlexPINN: Modeling Fluid Dynamics and Mass Transfer in 3D Micromixer Geometries Using a Flexible Physics-Informed Neural Network
- Dynamical gauge invariance of statistical mechanics
- ad-trait: A Fast and Flexible Automatic Differentiation Library in Rust
- Differentiable graph neural network simulator for forward and inverse modeling of multi-layered slope system with multiple material properties
- Deep Learning Meets Process-Based Models: A Hybrid Approach to Agricultural Challenges
- PyFRep: Shape Modeling with Differentiable Function Representation
- Safe Physics-Informed Machine Learning for Dynamics and Control
- Second-order Optimization of Gaussian Splats with Importance Sampling
- Towards asteroseismology of neutron stars with physics-informed neural networks
- BLAST: Bayesian online change-point detection with structured image data
- Level-set topology optimisation with unfitted finite elements and automatic shape differentiation
- Enabling Automatic Differentiation with Mollified Graph Neural Operators
- Comparison of deep learning software [wikipedia]
- Physics-informed neural networks [wikipedia]
Discussions
Related