Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

2025/06/06 by Motoki Omura, Omura, Motoki, Kazuki Ota +7
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2506.05968

openalex publication_date 2025/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continuous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overestimation bias. To address this, we propose an annealing approach that gradually transitions from the Bellman optimality operator to the Bellman operator, thereby accelerating learning while mitigating bias. Our method, combined with TD3 and SAC, significantly outperforms existing approaches across various locomotion and manipulation tasks, demonstrating improved performance and robustness to hyperparameters related to optimality. The code for this study is available at https://github.com/motokiomura/annealed-q-learning.

Citations

• Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution
• REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
• Extreme Q-Learning: MaxEnt RL without Entropy
• Solving Continuous Control via Q-learning
• Offline Reinforcement Learning with Implicit Q-Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Agent57: Outperforming the Atari Human Benchmark
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Curriculum Dropout
• Reinforcement Learning with Deep Energy-Based Policies
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Playing Atari with Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Related

• Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions
• Towards an Unified Structure for Reinforcement Learning: an Optimization Approach
• Bellman operator convergence enhancements in reinforcement learning algorithms
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Parameterized Projected Bellman Operator