Modularity benefits reinforcement learning agents with competing homeostatic drives

2022/04/13 by Zack Dulberg, Dulberg, Zack, Rachit Dubey +5 · 1 citation
Computer Science · Neuroscience · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural dynamics and brain function #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2204.06608

openalex publication_date 2022/04/13 · openalex created_date 2022/04/19 · openalex updated_date 2026/07/28

Abstract

The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.

Cited by

• Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

Related

• Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning
• Modular Multi-Objective Deep Reinforcement Learning with Decision Values
• Modular Networks Prevent Catastrophic Interference in Model-Based\n Multi-Task Reinforcement Learning
• Modular Continual Learning in a Unified Visual Environment
• Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning