Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning

2025/05/20 by Viet Anh Khoa Tran, Viet Anh Tran, Tran, Viet Anh Khoa +4 · 1 voice
Biochemistry, Genetics and Molecular Biology · Computer Science · #68T05 (primary) #68T07 #68T45 (secondary) #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Biological sciences #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #I.2.10 #I.2.6 #Machine Learning (cs.LG) #Neurons and Cognition (q-bio.NC) #cs.AI #cs.LG #q-bio.NC

paper · pdf · doi:10.48550/arxiv.2505.14125

openalex publication_date 2025/05/20 · arxiv published 2025/05/20 · openalex created_date 2025/10/10 · arxiv updated 2026/01/26 · openalex updated_date 2026/07/28

Abstract

Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize. Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task. We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision. We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss. Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it. Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels. Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.

Citations

• Unsupervised pretraining in biological neural networks
• Self-supervised predictive learning accounts for cortical layer-specificity
• Harnessing small projectors and multiple views for efficient vision pretraining
• Revisiting Supervision for Continual Representation Learning
• Plasticity-Optimized Complementary Networks for Unsupervised Continual Learning
• Learning beyond sensations: how dreams organize neuronal representations
• Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants
• Regularizing with Pseudo-Negatives for Continual Self-Supervised Learning
• SIESTA: Efficient Online Continual Learning with Sleep
• Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
• Patching open-vocabulary models by interpolating weights
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Architecture Matters in Continual Learning
• Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• On the Role of Neural Collapse in Transfer Learning
• Continually Learning Self-Supervised Representations with Projected Functional Regularization
• Self-Supervised Models are Continual Learners
• DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
• Representational Continuity for Unsupervised Continual Learning
• On the Opportunities and Risks of Foundation Models
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Co²L: Contrastive Continual Learning
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• VICReg: Variance-Invariance-Covariance Regularization for\n Self-Supervised Learning
• Emerging Properties in Self-Supervised Vision Transformers
• How Well Does Self-Supervised Pre-Training Perform with Streaming Data?
• Orthogonal Projection Loss
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Exploring Simple Siamese Representation Learning
• Exploring Simple Siamese Representation Learning
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• Online Continual Learning under Extreme Memory Constraints
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Supermasks in Superposition
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Bootstrap your own latent: A new approach to self-supervised Learning
• Supervised Contrastive Learning
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• A Simple Framework for Contrastive Learning of Visual Representations
• Maintaining Discrimination and Fairness in Class Incremental Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Kornia: an Open Source Differentiable Computer Vision Library for\n PyTorch
• Continual Unsupervised Representation Learning
• Contrastive Multiview Coding
• Contrastive Multiview Coding
• Large Scale Incremental Learning
• Learning to Remember: A Synaptic Plasticity Driven Framework for\n Continual Learning
• Experience Replay for Continual Learning
• Efficient Lifelong Learning with A-GEM
• End-to-End Incremental Learning
• Representation Learning with Contrastive Predictive Coding
• Taskonomy: Disentangling Task Transfer Learning
• Overcoming catastrophic forgetting with hard attention to the task
• Memory Aware Synapses: Learning what (not) to forget
• FiLM: Visual Reasoning with a General Conditioning Layer
• EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
• Gradient Episodic Memory for Continual Learning
• Learning multiple visual domains with residual adapters
• Continual Learning with Deep Generative Replay
• iCaRL: Incremental Classifier and Representation Learning
• Learning without Forgetting
• Deep Residual Learning for Image Recognition
• Describing Textures in the Wild
• Fine-Grained Visual Classification of Aircraft
• The decade of the dendritic NMDA spike
• Spatiotemporally Graded NMDA Spike/Plateau Potentials in Basal Dendrites of Neocortical Pyramidal Neurons
• Requirement of dendritic calcium spikes for induction of spike‐timing‐dependent synaptic plasticity
• NEURONAL CIRCUITS OF THE NEOCORTEX
• Catastrophic Forgetting, Rehearsal and Pseudorehearsal

Discussions

• This is research from the new Dendritic Learning Group at PGI-15 (‪@fz-juelich.de‬). A huge thanks to my supervisor Willem Wybo and our institute head Emre Neftci! 📄 Preprint: arxiv.org/abs/2505.1412 [bsky, 4 points, 1 comments]

Related

• Co²L: Contrastive Continual Learning
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• Memory-efficient Continual Learning with Neural Collapse Contrastive
• Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning
• Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks