Beyond Transformers for Function Learning

2023/04/19 by Simon Segert, Jonathan Cohen, Segert, Simon +1
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2304.09979

openalex publication_date 2023/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The ability to learn and predict simple functions is a key aspect of human intelligence. Recent works have started to explore this ability using transformer architectures, however it remains unclear whether this is sufficient to recapitulate the extrapolation abilities of people in this domain. Here, we propose to address this gap by augmenting the transformer architecture with two simple inductive learning biases, that are directly adapted from recent models of abstract reasoning in cognitive science. The results we report demonstrate that these biases are helpful in the context of large neural network models, as well as shed light on the types of inductive learning biases that may contribute to human abilities in extrapolation.

Related

• Inductive Biases for Deep Learning of Higher-Level Cognition
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
• On the Bias Against Inductive Biases
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks