Explanations are a Means to an End: Decision Theoretic Explanation Evaluation

2025/06/28 by Ziyang Guo, Berk Ustun, Hullman, Jessica +3 · 2 voices · 1 citation
Computer Science · Neuroscience · #Adversarial Robustness in Machine Learning #Embodied and Extended Cognition #Explainable Artificial Intelligence (XAI) #cs.AI #stat.ML

paper · pdf · doi:10.48550/arxiv.2506.22740

openalex publication_date 2025/06/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Explanations of model behavior are commonly evaluated via proxy properties weakly tied to the purposes explanations serve in practice. We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task. This approach yields three distinct estimands: 1) a theoretical benchmark that upperbounds achievable performance by any agent with the explanation, 2) a human-complementary value that quantifies the theoretically attainable value that is not already captured by a baseline human decision policy, and 3) a behavioral value representing the causal effect of providing the explanation to human decision-makers. We instantiate these definitions in a practical validation workflow, and apply them to assess explanation potential and interpret behavioral effects in human-AI decision support and mechanistic interpretability.

Citations

• Feature Responsiveness Scores: Model-Agnostic Explanations for Recourse
• When combinations of humans and AI are useful: A systematic review and meta-analysis
• Prediction Policy Problems
• Job Market Signaling
• Equivalent Comparisons of Experiments

Cited by

• Informative Post-Hoc Explanations Only Exist for Simple Functions

Discussions

• For XAI it’s often thought explanations help (boundedly rational) user “unlock” info in features for some decision. But no one says this, they say vaguer things like “supporting trust”. We lay out som [bsky, 30 points, 2 comments]
• The general idea joked about here formalized in decision-theoretic terms arxiv.org/abs/2506.22740 [bsky, 5 points, 1 comments]

Related

• In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making
• In search of verifiability: Explanations rarely enable complementary performance in AI‐advised decision making
• The What, the Why, and the How of Artificial Explanations in Automated Decision-Making
• On the Relationship Between Explanations, Fairness Perceptions, and Decisions
• A Survey of the Various Methodologies Towards making Artificial Intelligence More Explainable