2019/02/26 by Tom Everitt, Pedro A. Ortega, Everitt, Tom +6 · 1 citation
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Auction Theory and Applications #FOS: Computer and information sciences #Game Theory and Applications #I.2.6 #I.2.8 #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.1902.09980
Mostly superseded by arXiv:2102.01685
openalex publication_date 2019/02/26 · arxiv created 2022/01/20 · arxiv updated 2022/01/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using causal influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes can the agent have an incentivize to observe, and (2) which nodes can the agent have an incentivize to control? The answers tell us which information and influence points need extra protection. For example, we may want a classifier for job applications to not use the ethnicity of the candidate, and a reinforcement learning agent not to take direct control of its reward mechanism. Different algorithms and training paradigms can lead to different causal influence diagrams, so our method can be used to identify algorithms with problematic incentives and help in designing algorithms with better incentives.