Linear Gradient Prediction with Control Variates

2025/11/07 by Kamil Ciosek, Ciosek, Kamil, Nicolò Felicioni +3
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2511.05187

openalex publication_date 2025/11/07 · openalex created_date 2025/11/11 · openalex updated_date 2026/07/28

Abstract

We propose a new way of training neural networks, with the goal of reducing training cost. Our method uses approximate predicted gradients instead of the full gradients that require an expensive backward pass. We derive a control-variate-based technique that ensures our updates are unbiased estimates of the true gradient. Moreover, we propose a novel way to derive a predictor for the gradient inspired by the theory of the Neural Tangent Kernel. We empirically show the efficacy of the technique on a vision transformer classification task.

Citations

• Low Rank Gradients and Where to Find Them
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Characterizing the Spectrum of the NTK via a Power Series Expansion
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• On the Convergence of SGD with Biased Gradients
• On the Similarity between the Laplace and Neural Tangent Kernels
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Gradient Descent Happens in a Tiny Subspace
• The Reversible Residual Network: Backpropagation Without Storing Activations
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Decoupled Neural Interfaces using Synthetic Gradients
• Optimization Methods for Large-Scale Machine Learning
• Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming

Related

• Altering Backward Pass Gradients improves Convergence
• A Tutorial on Neural Networks and Gradient-free Training
• How to guess a gradient
• Towards Scalable Backpropagation-Free Gradient Estimation
• Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients