An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning

2023/06/27 by Sebastian Müller, Müller, Sebastian, Vanessa Toborek +9 · 7 citations
Computer Science · Decision Sciences · Engineering · Mathematics · Psychology · #Artificial Intelligence (cs.AI) #Artificial intelligence #Attribution #Comparability #Computer science #Data science #Empirical research #Engineering #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Forecasting Techniques and Applications #Hyperparameter #Machine Learning (cs.LG) #Machine learning #Management science #Mathematics #Metric (unit) #Operations management #Psychology #Selection (genetic algorithm) #Social psychology #Statistics #Stock Market Forecasting Methods

paper · pdf · doi:10.48550/arxiv.2306.15786

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Rashomon Effect describes the following phenomenon: for a given dataset there may exist many models with equally good performance but with different solution strategies. The Rashomon Effect has implications for Explainable Machine Learning, especially for the comparability of explanations. We provide a unified view on three different comparison scenarios and conduct a quantitative evaluation across different datasets, models, attribution methods, and metrics. We find that hyperparameter-tuning plays a role and that metric selection matters. Our results provide empirical support for previously anecdotal evidence and exhibit challenges for both scientists and practitioners.

Cited by

• Translating the Rashomon Effect to Sequential Decision-Making Tasks
• The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning
• Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithms
• FA(IR)2MA-GLVQ – A hidden-feature-bias mitigation approach for fairness in classification learning based on generalized matrix learning vector quantization
• Rashomon in the Streets: Explanation Ambiguity in Scene Understanding
• Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML
• The Anatomy of Evidence: An Investigation Into Explainable ICD Coding

Related

• Can we Agree? On the Rashōmon Effect and the Reliability of Post-Hoc Explainable AI
• Amazing Things Come From Having Many Good Models
• Revisiting Rashomon: A Comment on "The Two Cultures"
• Practical Attribution Guidance for Rashomon Sets
• A Path to Simpler Models Starts With Noise