Nested Learning: The Illusion of Deep Learning Architectures

2025/12/31 by Ali Behrouz, Meisam Razaviyayn, Peilin Zhong +1 · 1 voice · 2 citations
Computer Science · #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2512.24695

Abstract

Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improve, and find effective solutions. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own context flow. Through the lenses of NL, existing deep learning methods learns from data through compressing their own context flow, and in-context learning naturally emerges in large models. NL suggests a philosophy to design more expressive learning algorithms with more levels, resulting in higher-order in-context learning and potentially unlocking effective continual learning capabilities. We advocate for NL by presenting three core contributions: (1) Expressive Optimizers: We show that known gradient-based optimizers, such as Adam, SGD with Momentum, etc., are in fact associative memory modules that aim to compress the gradients' information (by gradient descent). Building on this insight, we present other more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Learning Module: Taking advantage of NL's insights on learning algorithms, we present a sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of long/short-term memory. Combining our self-modifying sequence model with the continuum memory system, we present a continual learning module, called Hope, showing promising results in language modeling, knowledge incorporation, and few-shot generalization tasks, continual learning, and long-context reasoning tasks.

Citations

• Higher-order Linear Attention
• Fast weight programming and linear transformers: from machine learning to neurobiology
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Cartridges: Lightweight and general-purpose long context representations via self-study
• ATLAS: Learning to Optimally Memorize the Context at Test Time
• I-Con: A Unifying Framework for Representation Learning
• It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
• RWKV-7 "Goose" with Expressive Dynamic State Evolution
• Test-time regression: a unifying framework for designing sequence models with associative memory
• FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training
• MiniMax-01: Scaling Foundation Models with Lightning Attention
• Titans: Learning to Memorize at Test Time
• Gated Delta Networks: Improving Mamba2 with Delta Rule
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
• Longhorn: State Space Models are Amortized Online Learners
• Learning to (Learn at Test Time): RNNs with Expressive Hidden States
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length
• xLSTM: Extended Long Short-Term Memory
• The Illusion of State in State-Space Models
• Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
• MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection
• Mechanistic Design and Scaling of Hybrid Architectures
• Simple linear attention language models balance the recall-throughput tradeoff
• Transformers, parallel computation, and logarithmic depth
• On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era
• Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks
• PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels
• Uncovering mesa-optimization algorithms in Transformers
• Retentive Network: A Successor to Transformer for Large Language Models
• RWKV: Reinventing RNNs for the Transformer Era
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
• GPT-4 Technical Report
• Symbolic Discovery of Optimization Algorithms
• An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models
• Liquid Structural State-Space Models
• Simplified State Space Layers for Sequence Modeling
• Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
• Training Compute-Optimal Large Language Models
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
• A Modern Self-Referential Weight Matrix That Learns to Modify Itself
• Generative Adversarial Networks
• Highly accurate protein structure prediction with AlphaFold
• Saturated Transformers are Constant-Depth Threshold Circuits
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Linear Transformers Are Secretly Fast Weight Programmers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• PIQA: Reasoning about Physical Commonsense in Natural Language
• On the Variance of the Adaptive Learning Rate and Beyond
• Metalearned Neural Memory
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Learning deep representations by mutual information estimation and maximization
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Shampoo: Preconditioned Stochastic Tensor Optimization
• signSGD: Compressed Optimisation for Non-Convex Problems
• Deep Learning Scaling is Predictable, Empirically
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• Neural Semantic Encoders
• The LAMBADA dataset: Word prediction requiring a broad discourse context
• Exponential expressivity in deep neural networks through transient chaos
• Stochastic Variance Reduction for Nonconvex Optimization
• Mastering the game of Go with deep neural networks and tree search
• Neural Machine Translation by Jointly Learning to Align and Translate
• On the Number of Linear Regions of Deep Neural Networks
• ImageNet classification with deep convolutional neural networks
• Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements
• Long Short-Term Memory
• Some Studies in Machine Learning Using the Game of Checkers
• Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
• Are Emergent Abilities of Large Language Models a Mirage?
• Simplified neuron model as a principal component analyzer

Cited by

• Toward Interpretable and Generalizable AI in Regulatory Genomics
• Contextual Agentic Memory is a Memo, Not True Memory

Discussions

• Y este es el paper actual en aras "Nested Learning: The Illusion of Deep Learning Architectures" (Behrouz, et al. 2026) https://arxiv.org/abs/2512.24695 [bsky, 1 points, 1 comments]

Related

• Dynamic Nested Hierarchies: Pioneering Self-Evolution in Machine Learning Architectures for Lifelong Intelligence
• Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
• Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• Opening the black box of language acquisition