Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving

2025/06/04 by Li Zeqiao, Yijing Wang, Zeqiao, Li +10
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2506.03568

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Autonomous driving promises significant advancements in mobility, road safety and traffic efficiency, yet reinforcement learning and imitation learning face safe-exploration and distribution-shift challenges. Although human-AI collaboration alleviates these issues, it often relies heavily on extensive human intervention, which increases costs and reduces efficiency. This paper develops a confidence-guided human-AI collaboration (C-HAC) strategy to overcome these limitations. First, C-HAC employs a distributional proxy value propagation method within the distributional soft actor-critic (DSAC) framework. By leveraging return distributions to represent human intentions C-HAC achieves rapid and stable learning of human-guided policies with minimal human interaction. Subsequently, a shared control mechanism is activated to integrate the learned human-guided policy with a self-learning policy that maximizes cumulative rewards. This enables the agent to explore independently and continuously enhance its performance beyond human guidance. Finally, a policy confidence evaluation algorithm capitalizes on DSAC's return distribution networks to facilitate dynamic switching between human-guided and self-learning policies via a confidence-based intervention function. This ensures the agent can pursue optimal policies while maintaining safety and performance guarantees. Extensive experiments across diverse driving scenarios reveal that C-HAC significantly outperforms conventional methods in terms of safety, efficiency, and overall performance, achieving state-of-the-art results. The effectiveness of the proposed method is further validated through real-world road tests in complex traffic conditions. The videos and code are available at: https://github.com/lzqw/C-HAC.

Citations

• Demonstrating specification gaming in reasoning models
• Learning from Active Human Involvement through Proxy Value Propagation
• Guarded Policy Optimization with Imperfect Online Demonstrations
• Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving
• Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization
• MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
• ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
• End-to-End Urban Driving by Imitating a Reinforcement Learning Coach
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Conservative Safety Critics for Exploration
• A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Conservative Q-Learning for Offline Reinforcement Learning
• Learning to Walk in the Real World with Minimal Human Effort
• Deep Reinforcement Learning for Autonomous Driving: A Survey
• Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles
• Learning a Decision Module by Imitating Driver's Control Behaviors
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Learning Reward Functions by Integrating Human Demonstrations and Preferences
• Exploring the Limitations of Behavior Cloning for Autonomous Driving
• Off-Policy Deep Reinforcement Learning without Exploration
• Scalable agent alignment via reward modeling: a research direction
• Batch Active Preference-Based Learning of Reward Functions
• HG-DAgger: Interactive Imitation Learning with Human Experts
• Learning to Drive in a Day
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Deep reinforcement learning from human preferences
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Related

• From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning
• Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving
• Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization
• Human-compatible driving partners through data-regularized self-play reinforcement learning
• Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving