Neural Priming for Sample-Efficient Adaptation

2023/06/16 by Matthew Wallingford, Wallingford, Matthew, Vivek Ramanujan +13 · 4 citations
Computer Science · Medicine · #AI in cancer detection #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2306.10191

openalex publication_date 2023/06/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks given few or no labeled examples. Presented with class names or unlabeled test samples, Neural Priming enables the model to recall and conditions its parameters on relevant data seen throughout pretraining, thereby priming it for the test distribution. Neural Priming can be performed at test time, even for pretraining datasets as large as LAION-2B. Performing lightweight updates on the recalled data significantly improves accuracy across a variety of distribution shift and transfer learning benchmarks. Concretely, in the zero-shot setting, we see a 2.45% improvement in accuracy on ImageNet and 3.81% accuracy improvement on average across standard transfer learning benchmarks. Further, using Neural Priming at inference to adapt to distribution shift, we see a 1.41% accuracy improvement on ImageNetV2. These results demonstrate the effectiveness of Neural Priming in addressing the challenge of limited labeled data and changing distributions. Code is available at github.com/RAIVNLab/neural-priming.

Cited by

• D3G: Diverse Demographic Data Generation Increases Zero-Shot Image Classification Accuracy within Multimodal Models
• Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective
• Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey

Related

• Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• Memory-based Parameter Adaptation
• Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation