The Elements of Differentiable Programming

2024/03/21 by Mathieu Blondel, Vincent Roulet, Blondel, Mathieu +1 · 10 voices · 14 citations
Engineering · #Advanced Control Systems Optimization #Optimization and Mathematical Programming #Scheduling and Optimization Algorithms #cs.AI #cs.LG #cs.PL

paper · pdf · doi:10.48550/arxiv.2403.14606

openalex publication_date 2024/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming. This new programming paradigm enables end-to-end differentiation of complex computer programs (including those with control flows and data structures), making gradient-based optimization of program parameters possible. As an emerging paradigm, differentiable programming builds upon several areas of computer science and applied mathematics, including automatic differentiation, graphical models, optimization and statistics. This book presents a comprehensive review of the fundamental concepts useful for differentiable programming. We adopt two main perspectives, that of optimization and that of probability, with clear analogies between the two. Differentiable programming is not merely the differentiation of programs, but also the thoughtful design of programs intended for differentiation. By making programs differentiable, we inherently introduce probability distributions over their execution, providing a means to quantify the uncertainty associated with program outputs.

Cited by

• Revisiting Sparse Matrix Coloring and Bicoloring
• NNN: Next-Generation Neural Networks for Marketing Measurement
• Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation
• Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
• EventQueues: Autodifferentiable spike event queues for brain simulation on AI accelerators
• RUBIX: Differentiable forward modelling of galaxy spectral data cubes for gradient-based parameter estimation
• Hierarchical Strategic Decision-Making in Layered Mobility Systems
• Wall Shear Stress Reconstruction from Concentration: Differentiable Physics and Physics-Informed Neural Networks
• <tt>Effort.jl</tt>: a fast and differentiable emulator for the Effective Field Theory of the Large Scale Structure of the Universe
• Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
• Neural non-canonical Hamiltonian dynamics for long-time simulations
• Per-example gradients: a new frontier for understanding and improving optimizers
• Frequentist Cosmological Constraints from Full-Shape Clustering Measurements in DESI DR1
• Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environments

Discussions

• The Elements of Differentiable Programming [hn, 131 points, 77 comments]
• Recommended! arxiv.org/abs/2403.14606 [bsky, 70 points, 2 comments]
• For the theoretical side, there’s no better resource than the 2024 book by Blondel and Roulet: arxiv.org/abs/2403.14606 For the practical side, my friend @willtebbutt.bsky.social wrote stellar docum [bsky, 56 points, 5 comments]
• "The Elements of Differentiable Programming" link: arxiv.org/abs/2403.14606 Basically: "autodiff - it's everywhere! what is it, and how do you use it?" seems like a good resource for anyone interes [bsky, 32 points, 0 comments]
• one possible way of thinking about probabilistic models is them being a soft relaxation on control flows. (taken from this book which makes for very interesting browsing! arxiv.org/abs/2403.146... ) [bsky, 8 points, 1 comments]
• The Elements of Differentiable Programming [lobsters, 8 points, 1 comments]
• Mathieu Blondel and Vincent Roulet have an excellent textbook with advanced topics here: arxiv.org/abs/2403.14606 - it covers really interesting tricks like smoothing max to autodiff through every kin [bsky, 4 points, 1 comments]
• 💯 arxiv.org/abs/2403.14606 proceedings.neurips.cc/paper/2021/f... [bsky, 3 points, 1 comments]
• Unfortunately not no. Maybe the closest I can think of is sections in this excellent book arxiv.org/abs/2403.14606, but it's a bit broader in scope. I wish there was something like @shakirm.bsky.socia [bsky, 3 points, 1 comments]
• The Elements of Differentiable Programming https://arxiv.org/abs/2403.14606 [bsky, 0 points, 0 comments]

Related

• Differentiable programming: Generalization, characterization and limitations of deep learning
• A Differentiable Programming System to Bridge Machine Learning and Scientific Computing
• Differentiable Functional Program Interpreters
• Using Differentiable Programming for Flexible Statistical Modeling
• Coarsening Optimization for Differentiable Programming