Understanding the Origin of Information-Seeking Exploration in\n Probabilistic Objectives for Control

2021/03/11 by Beren Millidge, Anil C. Seth, Millidge, Beren +4 · 1 citation
Computer Science · Economics, Econometrics and Finance · #Artificial Intelligence (cs.AI) #Complex Systems and Time Series Analysis #Computability, Logic, AI Algorithms #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2103.06859

openalex publication_date 2021/03/11 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

The exploration-exploitation trade-off is central to the description of\nadaptive behaviour in fields ranging from machine learning, to biology, to\neconomics. While many approaches have been taken, one approach to solving this\ntrade-off has been to equip or propose that agents possess an intrinsic\n'exploratory drive' which is often implemented in terms of maximizing the\nagents information gain about the world -- an approach which has been widely\nstudied in machine learning and cognitive science. In this paper we\nmathematically investigate the nature and meaning of such approaches and\ndemonstrate that this combination of utility maximizing and information-seeking\nbehaviour arises from the minimization of an entirely difference class of\nobjectives we call divergence objectives. We propose a dichotomy in the\nobjective functions underlying adaptive behaviour between \evidence\nobjectives, which correspond to well-known reward or utility maximizing\nobjectives in the literature, and \divergence objectives which instead\nseek to minimize the divergence between the agent's expected and desired\nfutures, and argue that this new class of divergence objectives could form the\nmathematical foundation for a much richer understanding of the exploratory\ncomponents of adaptive and intelligent action, beyond simply greedy utility\nmaximization.\n

Citations

• Humans use directed and random exploration to solve the explore–exploit dilemma.
• Active inference and agency: optimal control without cost functions
• AN INTEGRATIVE THEORY OF LOCUS COERULEUS-NOREPINEPHRINE FUNCTION: Adaptive Gain and Optimal Performance
• Approximate Inference and Stochastic Optimal Control
• Whatever next? Predictive brains, situated agents, and the future of cognitive science
• The Market for "Lemons": Quality Uncertainty and the Market Mechanism
• Exploration and Exploitation in Organizational Learning
• Planning and acting in partially observable stochastic domains
• Active Inference: A Process Theory
• Variational Intrinsic Control
• Active Inference, Curiosity and Insight
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Maximum a Posteriori Policy Optimisation
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Soft Actor-Critic Algorithms and Applications
• Efficient Exploration via State Marginal Matching
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Generalised free energy and active inference
• Reinforcement Learning through Active Inference
• Whence the Expected Free Energy?
• Deep active inference agents using Monte-Carlo methods
• Action and Perception as Divergence Minimization
• Online reinforcement learning with sparse rewards through an active inference capsule
• Relative Variational Intrinsic Control

Cited by

• Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective

Related

• Action and Perception as Divergence Minimization
• Humans use directed and random exploration to solve the explore–exploit dilemma.
• Knowledge is reward: Learning optimal exploration by predictive reward cashing
• Uncertainty quantification and exploration-exploitation trade-off in\n humans
• Information-Seeking Decision Strategies Mitigate Risk in Dynamic, Uncertain Environments