BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces

2024/10/28 by Landers, Matthew, Killian, Taylor W., Barnes, Hugo +2
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2410.21151

Abstract

Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose Branch Value Estimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to 20× in environments with over four million actions.

Related

• An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• Scalable Offline Model-Based RL with Action Chunks
• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
• Deep Reinforcement Learning in Large Discrete Action Spaces