NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation

2025/03/31 by Qinyu Li, Yee Whye Teh, Li, Qinyu +3 · 13 voices · 5 citations
Computer Science · #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2503.24322

Abstract

The canonical deep learning approach for learning requires computing a gradient term at each block by back-propagating the error signal from the output towards each learnable parameter. Given the stacked structure of neural networks, where each block builds on the representation of the block below, this approach leads to hierarchical representations. More abstract features live on the top blocks of the model, while features on lower blocks are expected to be less abstract. In contrast to this, we introduce a new learning method named NoProp, which does not rely on either forward or backwards propagation across the entire network. Instead, NoProp takes inspiration from diffusion and flow matching methods, where each block independently learns to denoise a noisy target using only local targets and back-propagation within the block. We believe this work takes a first step towards introducing a new family of learning methods that does not learn hierarchical representations -- at least not in the usual sense. NoProp needs to fix the representation at each block beforehand to a noised version of the target, learning a local denoising process that can then be exploited at inference. We demonstrate the effectiveness of our method on MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks. Our results show that NoProp is a viable learning algorithm, is easy to use and computationally efficient. By departing from the traditional learning paradigm which requires back-propagating a global error signal, NoProp alters how credit assignment is done within the network, enabling more efficient distributed learning as well as potentially impacting other characteristics of the learning process.

Citations

• Gradients without Backpropagation
• Neural Ordinary Differential Equations
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Cited by

• Pretraining Recurrent Networks without Recurrence
• Scaling Equilibrium Propagation to Deeper Neural Network Architectures
• Learning without Global Backpropagation via Synergistic Information Distillation
• Can Local Learning Match Self-Supervised Backpropagation?
• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis

Discussions

• NoProp: Training neural networks without back-propagation or forward-propagation [hn, 161 points, 49 comments]
• i don't understand this paper, but i don't understand it in the way that feels good and not the way that doesn't feel good arxiv.org/abs/2503.24322 [bsky, 39 points, 3 comments]
• [banging on the bottom of a pot with a spoon and marching around my house] i found a use for this absolutely wild paper arxiv.org/abs/2503.24322 [bsky, 37 points, 4 comments]
• NoProp: Training Neural Nets Without Backprop or Forwardprop A new paper proposes "NoProp"—a radical approach to training neural networks without traditional back/forward propagation. arxiv.org/abs/25 [bsky, 1 points, 0 comments]
• NoProp: Training neural networks without back-propagation or forward-propagation https://arxiv.org/abs/2503.24322 https://news.ycombinator.com/item?id=43676837 [bsky, 1 points, 0 comments]
• NoProp: Training neural networks without back-propagation or forward-propagation #ai #news [bsky, 0 points, 0 comments]
• NoProp: Training Neural Networks without Back-propagation or Forward-propagation https://arxiv.org/abs/2503.24322 [bsky, 0 points, 0 comments]
• NoProp: Training neural networks without back-propagation or forward-propagation https://arxiv.org/abs/2503.24322 [comments] [144 points] [bsky, 0 points, 0 comments]
• NoProp: Training Neural Networks without Back-propagation or Forward-propagation #HackerNews https://arxiv.org/abs/2503.24322 [bsky, 0 points, 0 comments]
• НетПроп: Обучение нейронных сетей без обратного распространения или прямого распространения #ai #news [bsky, 0 points, 0 comments]
• NoProp: Training Neural Networks without Back-propagation or Forward-propagation https://arxiv.org/abs/2503.24322 (https://news.ycombinator.com/item?id=43676837) [bsky, 0 points, 0 comments]
• NoProp: Training Neural Networks without Back-propagation or Forward-propagation https://arxiv.org/abs/2503.24322 (https://news.ycombinator.com/item?id=43676837) [bsky, 0 points, 0 comments]
• NoProp: Training Neural Networks without Back-propagation or Forward-propagation [bsky, 0 points, 0 comments]

Related

• The Cascaded Forward Algorithm for Neural Network Training
• Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses
• Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks
• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Blockwise Self-Supervised Learning at Scale