UACER: An Uncertainty-Adaptive Critic Ensemble Framework for Robust Adversarial Reinforcement Learning

2025/12/11 by Jiaxi Wu, Wu, Jiaxi, Tiantian Zhang +7
Computer Science · #Adaptive Dynamic Programming Control #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2512.10492

openalex publication_date 2025/12/11 · openalex created_date 2025/12/13 · openalex updated_date 2026/07/29

Abstract

Robust adversarial reinforcement learning has emerged as an effective paradigm for training agents to handle uncertain disturbance in real environments, with critical applications in sequential decision-making domains such as autonomous driving and robotic control. Within this paradigm, agent training is typically formulated as a zero-sum Markov game between a protagonist and an adversary to enhance policy robustness. However, the trainable nature of the adversary inevitably induces non-stationarity in the learning dynamics, leading to exacerbated training instability and convergence difficulties, particularly in high-dimensional complex environments. In this paper, we propose a novel approach, Uncertainty-Adaptive Critic Ensemble for robust adversarial Reinforcement learning (UACER), which consists of two components: 1) Diversified critic ensemble: A diverse set of K critic networks is employed in parallel to stabilize Q-value estimation in robust adversarial reinforcement learning, reducing variance and enhancing robustness compared to conventional single-critic designs. 2) Time-varying Decay Uncertainty (TDU) mechanism: Moving beyond simple linear combinations, we propose a variance-derived Q-value aggregation strategy that explicitly incorporates epistemic uncertainty to adaptively regulate the exploration-exploitation trade-off while stabilizing the training process. Comprehensive experiments across several challenging MuJoCo control problems validate the superior effectiveness of UACER, outperforming state-of-the-art methods in terms of overall performance, stability, and efficiency.

Citations

• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
• Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks
• Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
• Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space
• Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
• Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization
• Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
• Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark
• Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion
• Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization
• Ensemble Bootstrapping for Q-Learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep\n Reinforcement Learning
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Reinforcement Learning in Healthcare: A Survey
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Risk Averse Robust Adversarial Reinforcement Learning
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Parameter Space Noise for Exploration
• Robust Adversarial Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep\n Reinforcement Learning
• Deep Exploration via Bootstrapped DQN

Related

• Adversarially Trained Actor Critic for Offline Reinforcement Learning
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods
• Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method
• Risk Averse Robust Adversarial Reinforcement Learning
• Online Robust Multi-Agent Reinforcement Learning under Model Uncertainties