How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

2014/07/29 by Yoshua Bengio, Bengio, Yoshua · 13 citations
Business, Management and Accounting · Computer Science · #FOS: Computer and information sciences #Financial Distress and Bankruptcy Prediction #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.1407.7906

openalex publication_date 2014/07/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose to exploit \em reconstruction as a layer-local training signal for deep learning. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on derivatives in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can could thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.

Citations

• Generative Adversarial Networks
• Auto-Encoding Variational Bayes
• Connectionist learning of belief networks
• Slow Feature Analysis: Unsupervised Learning of Invariances

Cited by

• In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithm
• The brain as a blueprint: a survey of brain-inspired approaches to learning in artificial intelligence
• Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition
• Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks
• Dopamine-driven synaptic credit assignment in neural networks
• A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation
• Biological credit assignment through dynamic inversion of feedforward networks
• Biologically Plausible Learning via Bidirectional Spike-Based Distillation
• LoCo: Local Contrastive Representation Learning
• Adaptive Spatial Goodness Encoding: Advancing and Scaling Forward-Forward Learning Without Backpropagation
• Decoupling Search and Learning in Neural Net Training
• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Cortical credit assignment by Hebbian, neuromodulatory and inhibitory plasticity

Related

• Difference Target Propagation
• Conducting Credit Assignment by Aligning Local Representations
• Gradient-adjusted Incremental Target Propagation Provides Effective Credit Assignment in Deep Neural Networks
• Credit Assignment Through Broadcasting a Global Error Vector
• Memorization in Overparameterized Autoencoders