2024/01/14 by Xiao Liu, Jie Zhao, Liu, Xiao +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial intelligence #Black box #Computer science #Decision tree #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Fidelity #Focus (optics) #Machine Learning (cs.LG) #Machine learning #Process (computing) #Reinforcement learning #Set (abstract data type) #Software deployment #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2401.07263
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/01/14 · openalex created_date 2024/01/18 · openalex updated_date 2026/07/28
Despite the impressive capabilities of Deep Reinforcement Learning (DRL) agents in many challenging scenarios, their black-box decision-making process significantly limits their deployment in safety-sensitive domains. Several previous self-interpretable works focus on revealing the critical states of the agent's decision. However, they cannot pinpoint the error-prone states. To address this issue, we propose a novel self-interpretable structure, named Backbone Extract Tree (BET), to better explain the agent's behavior by identify the error-prone states. At a high level, BET hypothesizes that states in which the agent consistently executes uniform decisions exhibit a reduced propensity for errors. To effectively model this phenomenon, BET expresses these states within neighborhoods, each defined by a curated set of representative states. Therefore, states positioned at a greater distance from these representative benchmarks are more prone to error. We evaluate BET in various popular RL environments and show its superiority over existing self-interpretable models in terms of explanation fidelity. Furthermore, we demonstrate a use case for providing explanations for the agents in StarCraft II, a sophisticated multi-agent cooperative game. To the best of our knowledge, we are the first to explain such a complex scenarios using a fully transparent structure.