Deep active inference agents using Monte-Carlo methods

2020/06/07 by Zafeirios Fountas, Noor Sajid, Fountas, Zafeirios +5 · 5 citations
Computer Science · Neuroscience · #Artificial Intelligence (cs.AI) #Embodied and Extended Cognition #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (stat.ML) #Neural dynamics and brain function #Neurons and Cognition (q-bio.NC) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2006.04176

openalex publication_date 2020/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Active inference is a Bayesian framework for understanding biological intelligence. The underlying theory brings together perception and action under one single imperative: minimizing free energy. However, despite its theoretical utility in explaining intelligence, computational implementations have been restricted to low-dimensional and idealized situations. In this paper, we present a neural architecture for building deep active inference agents operating in complex, continuous state-spaces using multiple forms of Monte-Carlo (MC) sampling. For this, we introduce a number of techniques, novel to active inference. These include: i) selecting free-energy-optimal policies via MC tree search, ii) approximating this optimal policy distribution via a feed-forward `habitual' network, iii) predicting future parameter belief updates using MC dropouts and, finally, iv) optimizing state transition precision (a high-end form of attention). Our approach enables agents to learn environmental dynamics efficiently, while maintaining task performance, in relation to reward-based counterparts. We illustrate this in a new toy environment, based on the dSprites data-set, and demonstrate that active inference agents automatically create disentangled representations that are apt for modeling state transitions. In a more complex Animal-AI environment, our agents (using the same neural architecture) are able to simulate future state transitions and actions (i.e., plan), to evince reward-directed navigation - despite temporary suspension of visual input. These results show that deep active inference - equipped with MC methods - provides a flexible framework to develop biologically-inspired intelligent agents, with applications in both machine learning and cognitive science.

Citations

• Adam: A Method for Stochastic Optimization
• Active Inference: A Process Theory
• Deep active inference
• Disentangling by Factorising
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Bayesian policy selection using active inference
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Generalised free energy and active inference
• Control as Hybrid Inference
• Variational Inference: A Review for Statisticians

Cited by

• Prior preferences in active inference agents: soft, hard, and goal shaping
• Real-World Robot Control by Deep Active Inference With a Temporally Hierarchical World Model
• Zero-shot Structure Learning and Planning for Autonomous Robot Navigation using Active Inference
• Curiosity-Driven Development of Action and Language in Robots Through Self-Exploration
• A Computational Perspective on NeuroAI and Synthetic Biological Intelligence

Related

• Deconstructing deep active inference
• Deep Active Inference Agents for Delayed and Long-Horizon Environments
• Learning Perception and Planning with Deep Active Inference
• Simulating Biological Intelligence: Active Inference with Experiment-Informed Generative Model
• Deep Active Inference as Variational Policy Gradients