2020/11/12 by Sirui Xie, Feng Gao, Xie, Sirui +4
Computer Science · Mathematics · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Expected utility hypothesis #FOS: Computer and information sciences #Generalization #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine learning #Markov decision process #Markov process #Mathematics #Reinforcement Learning in Robotics #Reinforcement learning #Representation (politics) #Theoretical computer science #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2011.09854
published in arXiv (Cornell University) (Cornell University)
arxiv created 2020/11/12 · openalex publication_date 2020/11/12 · arxiv updated 2020/11/20 · openalex created_date 2020/11/23 · openalex updated_date 2026/07/28
In searching for a generalizable representation of temporally extended tasks, we spot two necessary constituents: the utility needs to be non-Markovian to transfer temporal relations invariant to a probability shift, the utility also needs to be lifted to abstract out specific grounding objects. In this work, we study learning such utility from human demonstrations. While inverse reinforcement learning (IRL) has been accepted as a general framework of utility learning, its fundamental formulation is one concrete Markov Decision Process. Thus the learned reward function does not specify the task independently of the environment. Going beyond that, we define a domain of generalization that spans a set of planning problems following a schema. We hence propose a new quest, Generalized Inverse Planning, for utility learning in this domain. We further outline a computational framework, Maximum Entropy Inverse Planning (MEIP), that learns non-Markovian utility and associated concepts in a generative manner. The learned utility and concepts form a task representation that generalizes regardless of probability shift or structural change. Seeing that the proposed generalization problem has not been widely studied yet, we carefully define an evaluation protocol, with which we illustrate the effectiveness of MEIP on two proof-of-concept domains and one challenging task: learning to fold from demonstrations.