Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control

2021/03/11 by Beren Millidge, Anil Seth, Anil C. Seth +6 · 1 citation
Computer Science · Economics, Econometrics and Finance · #Artificial Intelligence (cs.AI) #Complex Systems and Time Series Analysis #Computability, Logic, AI Algorithms #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2103.06859

11-03-21 initial upload. 14-03-21 fix Charnov citation. 16-03-21 another fix. 25-06-21 more fixes plus numerical simulations. 30-06-21 minor fixes; 12/11/21 maths typo fix; 24/11/21 minor maths fixes

openalex publication_date 2021/03/11 · arxiv created 2021/11/24 · arxiv updated 2021/11/29 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. While many approaches have been taken, one approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' which is often implemented in terms of maximizing the agents information gain about the world -- an approach which has been widely studied in machine learning and cognitive science. In this paper we mathematically investigate the nature and meaning of such approaches and demonstrate that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between evidence objectives, which correspond to well-known reward or utility maximizing objectives in the literature, and divergence objectives which instead seek to minimize the divergence between the agent's expected and desired futures, and argue that this new class of divergence objectives could form the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization.

Citations

• Humans use directed and random exploration to solve the explore–exploit dilemma.
• Active inference and agency: optimal control without cost functions
• AN INTEGRATIVE THEORY OF LOCUS COERULEUS-NOREPINEPHRINE FUNCTION: Adaptive Gain and Optimal Performance
• Approximate Inference and Stochastic Optimal Control
• Whatever next? Predictive brains, situated agents, and the future of cognitive science
• The Market for "Lemons": Quality Uncertainty and the Market Mechanism
• Exploration and Exploitation in Organizational Learning
• Planning and acting in partially observable stochastic domains
• Active Inference: A Process Theory
• Variational Intrinsic Control
• Active Inference, Curiosity and Insight
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Maximum a Posteriori Policy Optimisation
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Soft Actor-Critic Algorithms and Applications
• Efficient Exploration via State Marginal Matching
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Generalised free energy and active inference
• Reinforcement Learning through Active Inference
• Whence the Expected Free Energy?
• Deep active inference agents using Monte-Carlo methods
• Action and Perception as Divergence Minimization
• Online reinforcement learning with sparse rewards through an active inference capsule
• Relative Variational Intrinsic Control

Cited by

• Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective

Related

• Action and Perception as Divergence Minimization
• Humans use directed and random exploration to solve the explore–exploit dilemma.
• Knowledge is reward: Learning optimal exploration by predictive reward cashing
• Uncertainty quantification and exploration-exploitation trade-off in\n humans
• Information-Seeking Decision Strategies Mitigate Risk in Dynamic, Uncertain Environments