Is Optimal Transport Necessary for Inverse Reinforcement Learning?

2025/06/07 by Zixuan Dong, Dong, Zixuan, Keith W. Ross +2
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning

paper · pdf · doi:10.48550/arxiv.2506.06793

openalex publication_date 2025/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstrations. Recently, Optimal Transport (OT) methods have been successfully deployed to align trajectories and infer rewards. While OT-based methods have shown strong empirical results, they introduce algorithmic complexity, hyperparameter sensitivity, and require solving the OT optimization problems. In this work, we challenge the necessity of OT in IRL by proposing two simple, heuristic alternatives: (1) Minimum-Distance Reward, which assigns rewards based on the nearest expert state regardless of temporal order; and (2) Segment-Matching Reward, which incorporates lightweight temporal alignment by matching agent states to corresponding segments in the expert trajectory. These methods avoid optimization, exhibit linear-time complexity, and are easy to implement. Through extensive evaluations across 32 online and offline benchmarks with three reinforcement learning algorithms, we show that our simple rewards match or outperform recent OT-based approaches. Our findings suggest that the core benefits of OT may arise from basic proximity alignment rather than its optimal coupling formulation, advocating for reevaluation of complexity in future IRL design.

Citations

• Robot Policy Learning with Temporal Optimal Transport Reward
• Rethinking Optimal Transport in Offline Reinforcement Learning
• Align Your Intents: Offline Imitation Learning via Optimal Transport
• Imitation Learning from Observation with Automatic Discount Scheduling
• Pre-training with Synthetic Data Helps Offline Reinforcement Learning
• Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations
• CORL: Research-oriented Deep Offline Reinforcement Learning Library
• Watch and Match: Supercharging Imitation with Regularized Optimal Transport
• Offline Reinforcement Learning with Implicit Q-Learning
• Cross-Domain Imitation Learning via Optimal Transport
• Primal Wasserstein Imitation Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Wasserstein Adversarial Imitation Learning
• A survey of inverse reinforcement learning: Challenges, methods and progress

Related

• Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning
• Optimal Transport for Offline Imitation Learning
• Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
• Inverse Reinforcement Learning via Matching of Optimality Profiles
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective