Asking Easy Questions: A User-Friendly Approach to Active Reward\n Learning

2019/10/10 by Erdem Bıyık, Bıyık, Erdem, Malayandi Palan +7 · 7 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.1910.04365

openalex publication_date 2019/10/10 · openalex created_date 2022/07/28 · openalex updated_date 2026/08/04

Abstract

Robots can learn the right reward function by querying a human expert.\nExisting approaches attempt to choose questions where the robot is most\nuncertain about the human's response; however, they do not consider how easy it\nwill be for the human to answer! In this paper we explore an information gain\nformulation for optimally selecting questions that naturally account for the\nhuman's ability to answer. Our approach identifies questions that optimize the\ntrade-off between robot and human uncertainty, and determines when these\nquestions become redundant or costly. Simulations and a user study show our\nmethod not only produces easy questions, but also ultimately results in faster\nreward learning.\n

Citations

• Power to the People: The Role of Humans in Interactive Machine Learning
• An analysis of approximations for maximizing submodular set functions—I
• Bayesian Active Learning for Classification and Preference Learning
• A survey of robot learning from demonstration
• Information-Based Objective Functions for Active Data Selection
• An Active Learning Algorithm for Ranking from Pairwise Preferences with an Almost Optimal Query Complexity
• Near-Optimal Bayesian Active Learning with Noisy Observations

Cited by

• Efficient Personalization of Generative Models via Optimal Experimental Design
• Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration
• QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents
• Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback
• Learning Real-World Acrobatic Flight from Human Preferences
• Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds
• DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition

Related

• Here's What I've Learned: Asking Questions that Reveal Reward Learning
• Active Reward Learning from Online Preferences
• Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences
• Maximizing BCI Human Feedback using Active Learning
• What Questions Should Robots Be Able to Answer? A Dataset of User Questions for Explainable Robotics