SPACeR: Self-Play Anchoring with Centralized Reference Models

2025/10/20 by Wei-Jer Chang, Akshay Rangesh, Chang, Wei-Jer +11
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #I.2.6 #I.2.9 #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO) #Traffic control and management

paper · pdf · doi:10.48550/arxiv.2510.18060

openalex publication_date 2025/10/20 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Developing autonomous vehicles (AVs) requires not only safety and efficiency, but also realistic, human-like behaviors that are socially aware and predictable. Achieving this requires sim agent policies that are human-like, fast, and scalable in multi-agent settings. Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data, producing realistic policies. However, these models are computationally expensive, slow during inference, and struggle to adapt in reactive, closed-loop scenarios. In contrast, self-play reinforcement learning (RL) scales efficiently and naturally captures multi-agent interactions, but it often relies on heuristics and reward shaping, and the resulting policies can diverge from human norms. We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a centralized reference policy to guide decentralized self-play. The reference model provides likelihood rewards and KL divergence, anchoring policies to the human driving distribution while preserving RL scalability. Evaluated on the Waymo Sim Agents Challenge, our method achieves competitive performance with imitation-learned policies while being up to 10x faster at inference and 50x smaller in parameter size than large generative models. In addition, we demonstrate in closed-loop ego planning evaluation tasks that our sim agents can effectively measure planner quality with fast and scalable traffic simulation, establishing a new paradigm for testing autonomous driving policies.

Citations

• Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation
• Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations
• DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling
• Building reliable sim driving agents by scaling self-play
• Robust Autonomy Emerges from Self-Play
• Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models
• SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
• Learning to Drive via Asymmetric Self-Play
• Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving
• GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
• Human-compatible driving partners through data-regularized self-play reinforcement learning
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries
• Trajeglish: Traffic Modeling as Next-Token Prediction
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation
• Language-Guided Traffic Simulation via Scene-Level Diffusion
• Video Prediction Models as Rewards for Reinforcement Learning
• The Waymo Open Sim Agents Challenge
• imitation: Clean Imitation Learning Implementations
• Emerging Properties in Self-Supervised Vision Transformers
• Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• On the Utility of Learning about Humans for Human-AI Coordination
• Proximal Policy Optimization Algorithms
• Imitating Driver Behavior with Generative Adversarial Networks
• Congested traffic states in empirical observations and microscopic simulations

Related

• Human-compatible driving partners through data-regularized self-play reinforcement learning
• Human-like autonomy emerges from self-play and a pinch of human data
• MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research
• Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving