Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learning

2025/06/14 by Xabier Olaz, Olaz, Xabier
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Reinforcement Learning in Robotics #stochastic dynamics and bifurcation

paper · pdf · doi:10.48550/arxiv.2506.12366

openalex publication_date 2025/06/14 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Deep Reinforcement Learning (DRL) agents often exhibit intricate failure modes that are difficult to understand, debug, and learn from. This opacity hinders their reliable deployment in real-world applications. To address this critical gap, we introduce ``Ghost Policies,'' a concept materialized through Arvolution, a novel Augmented Reality (AR) framework. Arvolution renders an agent's historical failed policy trajectories as semi-transparent ``ghosts'' that coexist spatially and temporally with the active agent, enabling an intuitive visualization of policy divergence. Arvolution uniquely integrates: (1) AR visualization of ghost policies, (2) a behavioural taxonomy of DRL maladaptation, (3) a protocol for systematic human disruption to scientifically study failure, and (4) a dual-learning loop where both humans and agents learn from these visualized failures. We propose a paradigm shift, transforming DRL agent failures from opaque, costly errors into invaluable, actionable learning resources, laying the groundwork for a new research field: ``Failure Visualization Learning.''

Citations

• Toward Debugging Deep Reinforcement Learning Programs with RLExplorer
• Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes
• Solving Rubik's Cube with a Robot Hand
• Unity: A General Platform for Intelligent Agents
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Playing Atari with Deep Reinforcement Learning
• Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities

Related

• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields
• Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
• Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress
• Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning