Learning to (Learn at Test Time)

2023/10/20 by Yu Sun, Xinhao Li, Sun, Yu +15 · 2 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2310.13807

openalex publication_date 2023/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run.

Cited by

• Titans Revisited: A Lightweight Reimplementation and Critical Analysis of a Test-Time Memory Model
• End-to-End Test-Time Training for Long Context

Related

• The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
• Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
• Learning to Learn from Weak Supervision by Full Supervision
• A critical analysis of self-supervision, or what we can learn from a single image
• No Reason for No Supervision: Improved Generalization in Supervised Models