Behavior-Consistent Deep Reinforcement Learning

2026/05/20 by Marcel Hussing, Liv G. d'Aliberti, Claas Voelcker +2 · 1 voice
Computer Science · #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2605.21214

arxiv published 2026/05/20 · arxiv updated 2026/05/21

Abstract

Reinforcement learning (RL) often exhibits high variance across training runs, leading to unreliable performance and posing a major challenge to deployment in real-world domains. In this work, we address the challenge of cross-run policy divergence by formalizing the problem of behavior-consistent RL, where the objective is to obtain policies that are both high-performing and distributionally similar across training runs. Our key observation is that maximum-entropy RL provides a direct mechanism for controlling behavioral divergence by anchoring runs to a common (uniform) prior. We prove that, for Boltzmann policies, choosing the temperature proportional to Q-function disagreement bounds the pairwise KL divergence between the induced policies. However, we also show that naïvely increasing entropy might impair policy optimization while amplifying off-policy error. Building upon these observations, we propose Q-value Expectile Disagreement (QED), a state-dependent temperature schedule that uses double-critic disagreement as a single-run proxy for cross-run disagreement. Empirically, we demonstrate that across 18 continuous-control tasks, QED reduces across-run divergence by two orders of magnitude without sacrificing performance, resulting in a considerable reduction in return variance at modest sample-efficiency costs.

Citations

• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• Replicable Reinforcement Learning with Linear Function Approximation
• From Generative to Episodic: Sample-Efficient Replicable Reinforcement Learning
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning
• Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
• Towards General-Purpose Model-Free Reinforcement Learning
• SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
• MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
• Can we hop in general? A discussion of benchmark selection and design using the Hopper environment
• Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
• When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning
• Position: Benchmarking is Limited in Reinforcement Learning Research
• Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
• Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
• Disentangling the Causes of Plasticity Loss in Neural Networks
• Beyond Expected Return: Accounting for Policy Reproducibility when Evaluating Reinforcement Learning Algorithms
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Replicability in Reinforcement Learning
• Replicable Reinforcement Learning
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting
• Efficient Online Reinforcement Learning with Offline Data
• The Phenomenon of Policy Churn
• The Primacy Bias in Deep Reinforcement Learning
• Reproducibility in Learning
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• Measuring the Reliability of Reinforcement Learning Algorithms
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
• Reinforcement learning
• Control Regularization for Reduced Variance Reinforcement Learning
• Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments
• CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity
• A Theory of Regularized Markov Decision Processes
• Soft Actor-Critic Algorithms and Applications
• Off-Policy Deep Reinforcement Learning without Exploration
• Residual Reinforcement Learning for Robot Control
• Deep Reinforcement Learning and the Deadly Triad
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Reinforcement Learning that Matters
• The Uncertainty Bellman Equation and Exploration
• A Distributional Perspective on Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Deep Exploration via Randomized Value Functions
• Reinforcement Learning with Deep Energy-Based Policies
• Deep Exploration via Bootstrapped DQN
• Deep Reinforcement Learning with Double Q-learning
• Reinforcement Learning: An Introduction

Discussions

• This was a fun collaboration between Penn, Princeton, and UT Austin. 🔗 arxiv.org/abs/2605.21214 🧑‍🎓 @liv-daliberti.bsky.social @cvoelcker.bsky.social @ben-eysenbach.bsky.social @ericeaton.bsky.soci [bsky, 2 points, 0 comments]

Related

• Robust Policy Optimization in Deep Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• On The Presence of Double-Descent in Deep Reinforcement Learning
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• Dissecting Deep RL with High Update Ratios: Combatting Value Divergence