On Learning to Think: Algorithmic Information Theory for Novel\n Combinations of Reinforcement Learning Controllers and Recurrent Neural World\n Models

2015/11/30 by Juergen Schmidhuber, Schmidhuber, Juergen · 5 voices · 26 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #Exploit #Machine learning #Neural Networks and Applications #Recurrent neural network #Reinforcement Learning in Robotics #Reinforcement learning #Scratch #cs.AI #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1511.09249

published in arXiv (Cornell University) (Cornell University) · 36 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:1404.7828

arxiv created 2015/11/30 · openalex publication_date 2015/11/30 · arxiv updated 2015/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper addresses the general problem of reinforcement learning (RL) in\npartially observable environments. In 2013, our large RL recurrent neural\nnetworks (RNNs) learned from scratch to drive simulated cars from\nhigh-dimensional video input. However, real brains are more powerful in many\nways. In particular, they learn a predictive model of their initially unknown\nenvironment, and somehow use it for abstract (e.g., hierarchical) planning and\nreasoning. Guided by algorithmic information theory, we describe RNN-based AIs\n(RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending\nsequences of tasks, some of them provided by the user, others invented by the\nRNNAI itself in a curious, playful fashion, to improve its RNN-based world\nmodel. Unlike our previous model-building RNN-based RL machines dating back to\n1990, the RNNAI learns to actively query its model for abstract reasoning and\nplanning and decision making, essentially "learning to think." The basic ideas\nof this report can be applied to many other cases where one RNN-like system\nexploits the algorithmic information content of another. They are taken from a\ngrant proposal submitted in Fall 2014, and also explain concepts such as\n"mirror neurons." Experimental results will be described in separate papers.\n

Citations

• Learning Finite-State Controllers for Partially Observable Environments
• Building and Understanding Adaptive Systems: A Statistical/Numerical Approach to Factory Automation and Brain Research
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• On Computable Numbers, with an Application to the Entscheidungsproblem
• Polynomial Theory of Complex Systems
• Planning and acting in partially observable stochastic domains
• An Unsolvable Problem of Elementary Number Theory
• Deep Learning in Neural Networks: An Overview

Cited by

• CORAL: Learning Amyloid Fibril Ligand Docking with Cooperative Binding Rewards
• Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents
• Vidarc: Embodied Video Diffusion Model for Closed-loop Control
• WPT: World-to-Policy Transfer via Online World Model Distillation
• Deep Reinforcement Learning From Raw Pixels in Doom
• Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks
• A Step Toward World Models: A Survey on Robotic Manipulation
• One Big Net For Everything
• AI Agents as Universal Task Solvers
• Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognition
• On the role of planning in model-based deep reinforcement learning
• Unity: A General Platform for Intelligent Agents
• Artificial Intelligence and its Role in Near Future
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Continual Learning Using World Models for Pseudo-Rehearsal
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• A Survey of Deep Learning Techniques for Mobile Robot Applications
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Reference Free Platform Adaptive Locomotion for Quadrupedal Robots using a Dynamics Conditioned Policy
• A Brief Survey of Associations Between Meta-Learning and General AI
• Derivatives of Turing machines in Linear Logic
• A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
• Coupled Control, Structured Memory, and Verifiable Action in Agentic AI (SCRAT -- Stochastic Control with Retrieval and Auditable Trajectories): A Comparative Perspective from Squirrel Locomotion and Scatter-Hoarding
• Will We Ever Have Conscious Machines?

Discussions

• On Learning to Think: Algorithmic Information Theory [hn, 155 points, 3 comments]
• Algorithmic Information Theory for RL Controllers and RNN World Models [hn, 1 points, 0 comments]
• ......https://arxiv.org/abs/1511.09249 [bsky, 0 points, 0 comments]
• Re: deepseek arxiv.org/abs/1511.09249 [bsky, 0 points, 0 comments]
• ... "On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models" besprochen, das NNs nun das Denken beibringen [bsky, 0 points, 1 comments]

Related

• Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development
• World Models in Artificial Intelligence: Sensing, Learning, and Reasoning Like a Child
• On the Reliability and Generalizability of Brain-inspired Reinforcement Learning Algorithms
• One Big Net For Everything
• Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learning