Gradients without Backpropagation

2022/02/17 by Atılım Güneş Baydin, Baydin, Atılım Güneş, Barak A. Pearlmutter +7 · 3 voices · 20 citations
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Automatic differentiation #Backpropagation #Computation #Computer science #Descent direction #Directional derivative #Engineering #Gradient descent #Gradient method #Machine Learning and Algorithms #Machine Learning and ELM #Mathematical analysis #Mathematics #Mode (computer interface) #Neural Networks and Applications #Range (aeronautics) #acm:68T07 #cs.LG #msc:68T07 #stat.ML

paper · pdf · doi:10.48550/arxiv.2202.08587

published in arXiv (Cornell University) (Cornell University) · 10 pages, 6 figures

arxiv created 2022/02/17 · openalex publication_date 2022/02/17 · arxiv updated 2022/02/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic differentiation algorithms that also includes the forward mode. We present a method to compute gradients based solely on the directional derivative that one can compute exactly and efficiently via the forward mode. We call this formulation the forward gradient, an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent. We demonstrate forward gradient descent in a range of problems, showing substantial savings in computation and enabling training up to twice as fast in some cases.

Cited by

• Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs
• NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation
• SURE Guided Posterior Sampling: Trajectory Correction for Diffusion-Based Inverse Problems
• FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative
• NetworkFF: Unified Layer Optimization in Forward-Only Neural Networks
• Towards Scalable Backpropagation-Free Gradient Estimation
• Dimensional Type Systems and Deterministic Memory Management: Design-Time Semantic Preservation in Native Compilation
• More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
• TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models
• Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning
• Memory Savings at What Cost? A Study of Alternatives to Backpropagation
• Backpropagation-Free Metropolis-Adjusted Langevin Algorithm
• Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
• Decidable By Construction: Design-Time Verification for Trustworthy AI
• The Program Hypergraph: Multi-Way Relational Structure for Geometric Algebra, Spatial Compute, and Physics-Aware Compilation
• Polynomial Speedup in Diffusion Models with the Multilevel Euler-Maruyama Method
• Adaptive Domain Models: Bayesian Evolution, Warm Rotation, and Principled Training for Geometric and Neuromorphic AI

Discussions

• Gradients Without Backpropagation [hn, 87 points, 35 comments]
• Gradients Without Backpropagation [hn, 4 points, 0 comments]
• Gradients Without Backpropagation [hn, 2 points, 0 comments]

Related

• Optimization without Backpropagation
• Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
• Towards Scalable Backpropagation-Free Gradient Estimation
• On the complexity of nonsmooth automatic differentiation
• Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms