Shared sensitivity to data distribution during learning in humans and transformer networks

2025/12/23 by Jacques Pesnot Lerousseau, Christopher Summerfield · 1 citation
Computer Science · #Information Retrieval and Search Behavior #Domain Adaptation and Few-Shot Learning #Advanced Graph Neural Networks

paper · doi:10.1038/s41562-025-02359-3

Abstract

Do humans learn like transformers? We trained both humans (n = 530) and transformer networks on a rule learning task where they had to respond to a query in a sequence. At test, we measured 'in-context' learning (generalize the rule to novel queries) and 'in-weights' learning (recall past experiences from memory). Manipulating the diversity and redundancy of examples in the training distribution, we found that humans and transformer networks respond in very similar ways. In both types of learner, redundancy and diversity trade off in driving in-weights and in-context learning, respectively, whereas a composite distribution with a balanced mix of redundancy and diversity allows the two strategies to be used in tandem. However, we also found that while humans benefit from dynamic training schedules that emphasize diverse examples early, transformers do not. So, while the same data-distributional properties promote learning in humans and transformer networks, only people benefit from curricula.

Citations

• On the Dangers of Stochastic Parrots
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Transformers learn in-context by gradient descent
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.

Cited by

• A Disproof of Large Language Model Consciousness: The Necessity of Continual Learning for Consciousness
• Human curriculum learning of a cue combination task

Related

• Data Distributional Properties Drive Emergent In-Context Learning in Transformers
• Comparing Generalization in Learning with Limited Numbers of Exemplars: Transformer vs. RNN in Attractor Dynamics
• A distributional simplicity bias in the learning dynamics of transformers
• Toward Understanding In-context vs. In-weight Learning
• Analyzing limits for in-context learning