ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning
2022/07/15 by Shengchao Hu, Li Chen, Hu, Shengchao +9 · 116 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Reinforcement Learning in Robotics #Robotics and Sensor-Based Localization #cs.CV
paper · pdf · doi:10.48550/arxiv.2207.07601
ECCV 2022
arxiv created 2022/07/18 · arxiv updated 2022/07/19
Abstract
Many existing autonomous driving paradigms involve a multi-stage discrete pipeline of tasks. To better predict the control signals and enhance user safety, an end-to-end approach that benefits from joint spatial-temporal feature learning is desirable. While there are some pioneering works on LiDAR-based input or implicit design, in this paper we formulate the problem in an interpretable vision-based setting. In particular, we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously, which is called ST-P3. Specifically, an egocentric-aligned accumulation technique is proposed to preserve geometry information in 3D space before the bird's eye view transformation for perception; a dual pathway modeling is devised to take past motion variations into account for future prediction; a temporal-based refinement unit is introduced to compensate for recognizing vision-based elements for planning. To the best of our knowledge, we are the first to systematically investigate each part of an interpretable end-to-end vision-based autonomous driving system. We benchmark our approach against previous state-of-the-arts on both open-loop nuScenes dataset as well as closed-loop CARLA simulation. The results show the effectiveness of our method. Source code, model and protocol details are made publicly available at https://github.com/OpenPerceptionX/ST-P3.
Cited by
- MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving
- KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System
- WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving
- LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
- Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
- Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
- TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learning
- Spatial Retrieval Augmented Autonomous Driving
- TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search
- WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
- WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
- FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis
- ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding
- CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
- nuScenes Revisited: Progress and Challenges in Autonomous Driving
- OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
- RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
- SimScale: Learning to Drive via Real-World Simulation at Scale
- Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving
- LaGen: Towards Autoregressive LiDAR Scene Generation
- Map-World: Masked Action planning and Path-Integral World Model for Autonomous Driving
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
- WPT: World-to-Policy Transfer via Online World Model Distillation
- From Features to Reference Points: Lightweight and Adaptive Fusion for Cooperative Autonomous Driving
- CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
- ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts
- Embodied Cognition Augmented End2End Autonomous Driving
- HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving
- RoCA: Robust Cross-Domain End-to-End Autonomous Driving
- SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
- Dino-Diffusion Modular Designs Bridge the Cross-Domain Gap in Autonomous Parking
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
- VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Driving
- SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries
- DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
- Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models
- HYPE: Hybrid Planning with Ego Proposal-Conditioned Predictions
- Scalable Offline Metrics for Autonomous Driving
- Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
- From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving
- BEV-VLM: Trajectory Planning via Unified BEV Abstraction
- OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
- Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
- MAP: End-to-End Autonomous Driving with Map-Assisted Planning
- Embodied Navigation Foundation Model
- BEVTraj: Map-Free End-to-End Trajectory Prediction in Bird's-Eye View with Deformable Attention and Sparse Goal Proposals
- LLM-based Human-like Traffic Simulation for Self-driving Tests
- WAVE-DETR Multi-Modal Visible and Acoustic Real-Life Drone Detector
- OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
- Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
- OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
- KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
- 2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model
- CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
- To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
- LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
- EvoPSF: Online Evolution of Autonomous Driving Models via Planning-State Feedback
- ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving
- Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning
- ReSim: Reliable World Simulation for Autonomous Driving
- DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Model
- EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Driving
- I2V-GS: Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generation
- PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving
- Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning
- ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving
- Self-supervised Pretraining for Integrated Prediction and Planning of Automated Vehicles
- Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations
- Breaking Imitation Bottlenecks: Reinforced Diffusion Powers Diverse Trajectory Generation
- VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
- World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
- Epona: Autoregressive Diffusion World Model for Autonomous Driving
- Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving
- BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting
- SceneCrafter: Controllable Multi-View Driving Scene Editing
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
- ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
- NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- FocalAD: Local Motion Planning for End-to-End Autonomous Driving
- DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
- CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Driving
- RF4D:Neural Radar Fields for Novel View Synthesis in Outdoor Dynamic Scenes
- DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy
- Echo Planning for Autonomous Driving: From Current Observations to Future Trajectories and Back
- Extended Field of View Analysis for VideoGAN-based Trajectory Generation
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
- RealEngine: Simulating Autonomous Driving in Realistic Context
- ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Driving
- VERDI: VLM-Embedded Reasoning for Autonomous Driving
- iPad: Iterative Proposal-centric End-to-End Autonomous Driving
- Generative AI for Autonomous Driving: A Review
- Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving
- M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark
- DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning
- PADriver: Towards Personalized Autonomous Driving
- UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
- Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning
- Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation
- LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
- Toward Fully Autonomous Driving: AI, Challenges, Opportunities, and Needs
- TwinIR: Coordinated Invisible Dual-Point Attacks on Online HD Map Construction
- Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction
- UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertainty
- Fully Unified Motion Planning for End-to-End Autonomous Driving
- Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Features
- E2E Parking Dataset: An Open Benchmark for End-to-End Autonomous Parking
Related