Counterfactual state explanations for reinforcement learning agents via generative deep learning

2021/01/27 by Matthew Olson, Matthew L. Olson, Roli Khanna +4 · 66 citations
Computer Science · Economics, Econometrics and Finance · Psychology · #Algorithm #Artificial intelligence #Baseline (sea) #Computer science #Counterfactual thinking #Explainable Artificial Intelligence (XAI) #Generative grammar #Generative model #Machine learning #Psychology #Reinforcement Learning in Robotics #Reinforcement learning #Social psychology #Sports Analytics and Performance #State (computer science) #cs.AI #cs.HC #cs.LG

paper · pdf · doi:10.1016/j.artint.2021.103455

published in Artificial Intelligence 295, 103455 (Elsevier BV) · Full source code available at https://github.com/mattolson93/counterfactual-state-explanations

openalex publication_date 2021/01/27 · arxiv created 2021/01/29 · arxiv updated 2021/02/01 · openalex created_date 2021/02/01 · openalex updated_date 2026/08/05

Abstract

Counterfactual explanations, which deal with "why not?" scenarios, can provide insightful explanations to an AI agent's behavior. In this work, we focus on generating counterfactual explanations for deep reinforcement learning (RL) agents which operate in visual input environments like Atari. We introduce counterfactual state explanations, a novel example-based approach to counterfactual explanations based on generative deep learning. Specifically, a counterfactual state illustrates what minimal change is needed to an Atari game image such that the agent chooses a different action. We also evaluate the effectiveness of counterfactual states on human participants who are not machine learning experts. Our first user study investigates if humans can discern if the counterfactual state explanations are produced by the actual game or produced by a generative deep learning approach. Our second user study investigates if counterfactual state explanations can help non-expert participants identify a flawed agent; we compare against a baseline approach based on a nearest neighbor explanation which uses images from the actual game. Our results indicate that counterfactual state explanations have sufficient fidelity to the actual game images to enable non-experts to more effectively identify a flawed RL agent compared to the nearest neighbor baseline and to having no explanation at all.

Citations

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Intelligible Models for HealthCare
• Striving for Simplicity: The All Convolutional Net
• Three Approaches to Qualitative Content Analysis
• Human-level control through deep reinforcement learning
• Representation Learning: A Review and New Perspectives
• Axiomatic Attribution for Deep Networks
• Understanding Black-box Predictions via Influence Functions
• Fader Networks: Manipulating Images by Sliding Attributes
• Visualizing and Understanding Atari Agents
• Enabling Robots to Communicate Their Objectives
• Sanity Checks for Saliency Maps
• Explaining Image Classifiers by Counterfactual Generation
• Learning Finite State Representations of Recurrent Policy Networks
• Model-Based Reinforcement Learning for Atari
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Counterfactual States for Atari Agents via Generative Deep Learning
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• Adversarial Autoencoders
• Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
• Towards Interpretable Reinforcement Learning Using Attention Augmented Agents
• Generative Adversarial Nets
• OpenAI Gym