Structured Imitation Learning of Interactive Policies through Inverse Games

2025/11/17 by Muchen Sun, Sun, Max M., Todd D. Murphey +1
Computer Science · Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2511.12848

openalex publication_date 2025/11/17 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

Generative model-based imitation learning methods have recently achieved strong results in learning high-complexity motor skills from human demonstrations. However, imitation learning of interactive policies that coordinate with humans in shared spaces without explicit communication remains challenging, due to the significantly higher behavioral complexity in multi-agent interactions compared to non-interactive tasks. In this work, we introduce a structured imitation learning framework for interactive policies by combining generative single-agent policy learning with a flexible yet expressive game-theoretic structure. Our method explicitly separates learning into two steps: first, we learn individual behavioral patterns from multi-agent demonstrations using standard imitation learning; then, we structurally learn inter-agent dependencies by solving an inverse game problem. Preliminary results in a synthetic 5-agent social navigation task show that our method significantly improves non-interactive policies and performs comparably to the ground truth interactive policy using only 50 demonstrations. These results highlight the potential of structured imitation learning in interactive settings.

Citations

• Autonomous Human-Robot Interaction via Operator Imitation
• Inverse Mixed Strategy Games with Generative Trajectory Models
• Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
• HumanPlus: Humanoid Shadowing and Imitation from Humans
• Mixed Strategy Nash Equilibrium for Crowd Navigation
• Mixed strategy Nash equilibrium for crowd navigation
• BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay
• Learning Human-to-Robot Handovers from Point Clouds
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Learning Agile Robotic Locomotion Skills by Imitating Animals
• Efficient Iterative Linear-Quadratic Approximations for Nonlinear\n Multi-Player General-Sum Differential Games
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Robot navigation in dense human crowds: Statistical models and experimental studies of human–robot cooperation

Related

• Multi-Agent Generative Adversarial Imitation Learning
• Conditional Imitation Learning for Multi-Agent Games
• Generative Adversarial Imitation Learning
• Error Bounds of Imitating Policies and Environments
• Imitation Learning: Progress, Taxonomies and Challenges