Cooperative Inverse Reinforcement Learning

2016/06/09 by Dylan Hadfield-Menell, Hadfield-Menell, Dylan, Anca Dragan +6 · 2 voices · 326 citations
Business, Management and Accounting · Computer Science · Decision Sciences · Mathematics · #Artificial intelligence #Auction Theory and Applications #Computer science #Isolation (microbiology) #Machine learning #Mathematical optimization #Mathematics #Maximization #Reinforcement Learning in Robotics #Reinforcement learning #Supply Chain and Inventory Management #Value (mathematics) #cs.AI

paper · pdf · doi:10.48550/arxiv.1606.03137

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2016/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.

Citations

• The Complexity of Decentralized Control of Markov Decision Processes
• Theory of the firm: Managerial behavior, agency costs and ownership structure

Cited by

• Supervised Reward Inference
• Estimating cognitive biases with attention-aware inverse planning
• AssistanceZero: Scalably Solving Assistance Games
• Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
• Why human-AI relationships need socioaffective alignment
• A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
• When Robots Say No: The Empathic Ethical Disobedience Benchmark
• Embedded Safety-Aligned Intelligence via Differentiable Internal Alignment Embeddings
• Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants
• A Decision-Theoretic Approach for Managing Misalignment
• Collaborative Causal Sensemaking: Closing the Complementarity Gap in Human-AI Decision Support
• Password-Activated Shutdown Protocols for Misaligned Frontier Agents
• The Specification Trap: Why Static Value Alignment Alone Is Insufficient for Robust Alignment
• Some economics of artificial superintelligence
• Indicating Robot Vision Capabilities with Augmented Reality
• The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy
• Pragmatic Reasoning in Design
• A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
• Training LLM Agents to Empower Humans
• From Delegates to Trustees: How Optimizing for Long-Term Interests Shapes Bias and Alignment in LLM
• Moral Anchor System: A Predictive Framework for AI Value Alignment and Drift Prevention
• AI Safety, Alignment, and Ethics (AI SAE)
• Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Response
• Is GPT-4o mini Blinded by its Own Safety Filters? Exposing the Multimodal-to-Unimodal Bottleneck in Hate Speech Detection
• Co-Alignment: Rethinking Alignment as Bidirectional Human-AI Cognitive Adaptation
• Inference of Altruism and Intrinsic Rewards in Multi-Agent Systems
• Embodied AI: Emerging Risks and Opportunities for Policy Action
• NiceWebRL: a Python library for human subject experiments with reinforcement learning environments
• Governance-as-a-Service: A Multi-Agent Framework for AI System Compliance and Policy Enforcement
• Open-Universe Assistance Games
• Critical Risks and Opportunities of AI, Economics, Social Network Theory
• Towards Integrated Alignment
• NPO: Learning Alignment and Meta-Alignment through Structured Human Feedback
• Quantum Algorithm for Apprenticeship Learning
• Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests
• Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates
• The Singapore Consensus on Global AI Safety Research Priorities
• Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)
• Constitutive Components for Human-Like Autonomous Artificial Intelligence
• Misalignment or misuse? The AGI alignment tradeoff
• Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models
• Balancing Explicability and Explanation in Human-Aware Planning
• Avoiding Side Effects By Considering Future Tasks
• Sequential Cooperative Bayesian Inference
• Mitigating Deceptive Alignment via Self-Monitoring
• OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Direct Preference Optimization for Adaptive Concept-based Explanations
• Assisted Robust Reward Design
• Deep Reinforcement Learning: An Overview
• Optimal Interactive Learning on the Job via Facility Location Planning
• Enabling Value Sensitive AI Systems through Participatory Design Fictions
• Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem
• Super Co-alignment of Human and AI for Sustainable Symbiotic Society
• Learning a commonsense moral theory
• Advanced artificial agents intervene in the provision of reward
• Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
• Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning
• AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection
• Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge
• Contemplative Agent
• Recursive Deep Inverse Reinforcement Learning

Discussions

• This week's #PaperILike is "Cooperative Inverse Reinforcement Learning" (Hadfield-Menell et al., 2016). Seminal work. My favorite part is the simple example showing that demonstration-by-expert is sub [bsky, 23 points, 0 comments]
• Cooperative Inverse Reinforcement Learning (2016) [hn, 6 points, 0 comments]

Related

• An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning
• Interactive Inverse Reinforcement Learning for Cooperative Games
• Non-Cooperative Inverse Reinforcement Learning
• Multi-Principal Assistance Games
• Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games