Putting An End to End-to-End: Gradient-Isolated Learning of Representations

2019/05/28 by Sindy Löwe, Löwe, Sindy, Peter O’Connor +3 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1905.11786

openalex publication_date 2019/05/28 · openalex created_date 2019/11/22 · openalex updated_date 2026/07/28

Abstract

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

Citations

• Towards Biologically Plausible Deep Learning
• Adam: A Method for Stochastic Optimization
• Kaldi Speech Recognition Toolkit
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Fast Learning Algorithm for Deep Belief Nets
• Slow Feature Analysis: Unsupervised Learning of Invariances
• Variational Graph Auto-Encoders
• Auto-Encoding Total Correlation Explanation
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Training Neural Networks with Local Error Signals
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Deep Residual Learning for Image Recognition

Cited by

• Can Local Learning Match Self-Supervised Backpropagation?
• A general framework for interpretable neural learning based on local information-theoretic goal functions
• ConFuse: Convolutional Transform Learning Fusion Framework For\n Multi-Channel Data Analysis
• MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Tasks

Related

• Can Local Learning Match Self-Supervised Backpropagation?
• LoCo: Local Contrastive Representation Learning
• Blockwise Self-Supervised Learning at Scale
• Gradients as Features for Deep Representation Learning
• Learning deep representations by mutual information estimation and maximization