2021/11/12 by Lawrence Chan, Lawrence S. Chan, Andrew Critch +5 · 2 citations
Arts and Humanities · Computer Science · #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Machine Learning (cs.LG) #Philosophy and History of Science #Reinforcement Learning in Robotics #cs.LG
paper · pdf · doi:10.48550/arxiv.2111.06956
12 pages, 10 figures
arxiv created 2021/11/12 · openalex publication_date 2021/11/12 · arxiv updated 2021/11/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior. But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference. The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization. We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference. We find that wrongly modeling a systematically irrational human as noisy-rational performs a lot worse than correctly capturing these biases -- so much so that it can be better to skip inference altogether and stick to the prior! More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can. That is, if a robot has the correct model of a human's irrationality, it can make an even stronger inference than it ever could if the human were rational. Irrationality fundamentally helps rather than hinder reward inference, but it needs to be correctly accounted for.