Putting An End to End-to-End: Gradient-Isolated Learning of Representations

2019/05/28 by Sindy Löwe, Peter O'Connor, Löwe, Sindy +4 · 1 voice · 50 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Artificial intelligence #Artificial neural network #Backpropagation #Computer science #Deep learning #Deep neural networks #Domain Adaptation and Few-Shot Learning #End-to-end principle #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Neural Networks and Applications #Pattern recognition (psychology) #Representation (politics) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1905.11786

published in arXiv (Cornell University) (Cornell University) · Honorable Mention for Outstanding New Directions Paper Award at NeurIPS 2019

openalex publication_date 2019/05/28 · arxiv published 2019/05/28 · openalex created_date 2019/11/22 · arxiv created 2020/01/27 · arxiv updated 2020/01/28 · openalex updated_date 2026/07/28

Abstract

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

Citations

• Towards Biologically Plausible Deep Learning
• Adam: A Method for Stochastic Optimization
• Kaldi Speech Recognition Toolkit
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Fast Learning Algorithm for Deep Belief Nets
• Slow Feature Analysis: Unsupervised Learning of Invariances
• Variational Graph Auto-Encoders
• Auto-Encoding Total Correlation Explanation
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Training Neural Networks with Local Error Signals
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Deep Residual Learning for Image Recognition

Cited by

• Can Local Learning Match Self-Supervised Backpropagation?
• A general framework for interpretable neural learning based on local information-theoretic goal functions
• ConFuse: Convolutional Transform Learning Fusion Framework For Multi-Channel Data Analysis
• MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Tasks
• Meta-Learning Bidirectional Update Rules
• Single Snapshot Distillation for Phase Coded Mask Design in Phase Retrieval
• Stochastic Forward-Forward Learning through Representational Dimensionality Compression
• Think Global, Act Local: Relating DNN generalisation and node-level SNR
• Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects
• The Forward-Forward Algorithm: Characterizing Training Behavior

Discussions

• Putting an End to End-to-End: Gradient-Isolated Learning of Representations [hn, 2 points, 0 comments]

Related

• Can Local Learning Match Self-Supervised Backpropagation?
• LoCo: Local Contrastive Representation Learning
• Blockwise Self-Supervised Learning at Scale
• Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data
• Gradients as Features for Deep Representation Learning