ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

2025/06/09 by Yongkang Li, Y. L. Li, Kaixin Xiong +29 · 1 voice · 44 citations
Computer Science · Engineering · #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robotics (cs.RO) #Transportation and Mobility Innovations #cs.CV #cs.RO

paper · pdf · doi:10.48550/arxiv.2506.08052

openalex publication_date 2025/06/09 · arxiv published 2025/06/09 · arxiv updated 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Recent studies have explored leveraging the world knowledge and cognitive capabilities of Vision-Language Models (VLMs) to address the long-tail problem in end-to-end autonomous driving. However, existing methods typically formulate trajectory planning as a language modeling task, where physical actions are output in the language space, potentially leading to issues such as format-violating outputs, infeasible actions, and slow inference speeds. In this paper, we propose ReCogDrive, a novel Reinforced Cognitive framework for end-to-end autonomous Driving, unifying driving understanding and planning by integrating an autoregressive model with a diffusion planner. First, to instill human driving cognition into the VLM, we introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers through three stages: generation, refinement, and quality control. Building on this cognitive foundation, we then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner to efficiently generate continuous and stable trajectories. Furthermore, to enhance driving safety and reduce collisions, we introduce a Diffusion Group Relative Policy Optimization (DiffGRPO) stage, reinforcing the planner for enhanced safety and comfort. Extensive experiments on the NAVSIM and Bench2Drive benchmarks demonstrate that ReCogDrive achieves state-of-the-art performance. Additionally, qualitative results across diverse driving scenarios and DriveBench highlight the model's scene comprehension. All code, model weights, and datasets will be made publicly available to facilitate subsequent research.

Citations

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
• Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving
• LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
• ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Driving
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• End-to-End Driving with Online Trajectory Evaluation via BEV World Model
• ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
• HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model
• AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning
• GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving
• DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving
• CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Driving
• Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning
• Qwen2.5-VL Technical Report
• RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Diffusion-Based Planning for Autonomous Driving with Flexible Guidance
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
• Open-Sora: Democratizing Efficient Video Production for All
• DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
• OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
• Qwen2.5 Technical Report
• WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model
• Phi-4 Technical Report
• DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving
• Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
• GPT-4o System Card
• RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
• Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning
• Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Diffusion Policy Policy Optimization
• DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mamba
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
• Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
• Instruct Large Language Models to Drive like Humans
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
• Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases
• SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving
• VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• DriveLM: Driving with Graph Visual Question Answering
• LingoQA: Visual Question Answering for Autonomous Driving
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models
• Dolphins: Multimodal Language Model for Driving
• A Language Agent for Autonomous Driving
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
• GPT-Driver: Learning to Drive with GPT
• Far3D: Expanding the Horizon for Surround-view 3D Object Detection
• DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Language-Guided Traffic Simulation via Scene-Level Diffusion
• NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
• Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes
• Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving
• Visual Instruction Tuning
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Mastering Diverse Domains through World Models
• Planning-oriented Autonomous Driving
• Scalable Diffusion Models with Transformers
• Guided Conditional Diffusion for Controllable Traffic Simulation
• DRAMA: Joint Risk Localization and Captioning in Driving
• ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries
• ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
• TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events
• Improved Denoising Diffusion Probabilistic Models
• Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
• Denoising Diffusion Probabilistic Models
• GLU Variants Improve Transformer
• Social Attention for Autonomous Decision-Making in Dense Traffic
• Root Mean Square Layer Normalization
• MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction
• nuScenes: A multimodal dataset for autonomous driving
• CARLA: An Open Urban Driving Simulator
• Learning Personalized Driving Styles via Reinforcement Learning from Human Feedback
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

Cited by

• World Engine: Towards the Era of Post-Training for Autonomous Driving
• ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness
• MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving
• SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
• Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
• DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
• UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
• InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
• TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learning
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving
• Spatial Retrieval Augmented Autonomous Driving
• WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
• E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
• SimScale: Learning to Drive via Real-World Simulation at Scale
• AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
• DAP: A Discrete-token Autoregressive Planner for Autonomous Driving
• FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring
• Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
• DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
• DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models
• CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving
• DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
• Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement
• MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases
• Driving on Registers
• DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
• AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
• A Survey on Vision-Language-Action Models for Autonomous Driving
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
• From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
• Unleashing the Potential of Diffusion Models for End-to-End Autonomous Driving
• Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments
• Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
• PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning

Discussions

• www.arxiv.org/abs/2506.08052 [bsky, 0 points, 0 comments]

Related

• VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• Less is More: Lean yet Powerful Vision-Language Model for Autonomous Driving
• WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model
• InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving