On Learning to Think: Algorithmic Information Theory for Novel\n Combinations of Reinforcement Learning Controllers and Recurrent Neural World\n Models

2015/11/30 by Juergen Schmidhuber, Schmidhuber, Juergen · 5 voices · 15 citations
Computer Science · #Neural Networks and Applications #Reinforcement Learning in Robotics #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.1511.09249

Abstract

This paper addresses the general problem of reinforcement learning (RL) in\npartially observable environments. In 2013, our large RL recurrent neural\nnetworks (RNNs) learned from scratch to drive simulated cars from\nhigh-dimensional video input. However, real brains are more powerful in many\nways. In particular, they learn a predictive model of their initially unknown\nenvironment, and somehow use it for abstract (e.g., hierarchical) planning and\nreasoning. Guided by algorithmic information theory, we describe RNN-based AIs\n(RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending\nsequences of tasks, some of them provided by the user, others invented by the\nRNNAI itself in a curious, playful fashion, to improve its RNN-based world\nmodel. Unlike our previous model-building RNN-based RL machines dating back to\n1990, the RNNAI learns to actively query its model for abstract reasoning and\nplanning and decision making, essentially "learning to think." The basic ideas\nof this report can be applied to many other cases where one RNN-like system\nexploits the algorithmic information content of another. They are taken from a\ngrant proposal submitted in Fall 2014, and also explain concepts such as\n"mirror neurons." Experimental results will be described in separate papers.\n

Citations

• Learning Finite-State Controllers for Partially Observable Environments
• Building and Understanding Adaptive Systems: A Statistical/Numerical Approach to Factory Automation and Brain Research
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• On Computable Numbers, with an Application to the Entscheidungsproblem
• Polynomial Theory of Complex Systems
• Planning and acting in partially observable stochastic domains
• An Unsolvable Problem of Elementary Number Theory

Cited by

• CORAL: Learning Amyloid Fibril Ligand Docking with Cooperative Binding Rewards
• Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents
• Vidarc: Embodied Video Diffusion Model for Closed-loop Control
• WPT: World-to-Policy Transfer via Online World Model Distillation
• Deep Reinforcement Learning From Raw Pixels in Doom
• Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks
• A Step Toward World Models: A Survey on Robotic Manipulation
• One Big Net For Everything
• AI Agents as Universal Task Solvers
• Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognition
• On the role of planning in model-based deep reinforcement learning
• Unity: A General Platform for Intelligent Agents
• Artificial Intelligence and its Role in Near Future
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Continual Learning Using World Models for Pseudo-Rehearsal

Discussions

• On Learning to Think: Algorithmic Information Theory [hn, 155 points, 3 comments]
• Algorithmic Information Theory for RL Controllers and RNN World Models [hn, 1 points, 0 comments]
• ......https://arxiv.org/abs/1511.09249 [bsky, 0 points, 0 comments]
• Re: deepseek arxiv.org/abs/1511.09249 [bsky, 0 points, 0 comments]
• ... "On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models" besprochen, das NNs nun das Denken beibringen [bsky, 0 points, 1 comments]

Related

• Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development
• World Models in Artificial Intelligence: Sensing, Learning, and Reasoning Like a Child
• On the Reliability and Generalizability of Brain-inspired Reinforcement Learning Algorithms
• One Big Net For Everything
• Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learning