Does the Model Say What the Data Says? A Simple Heuristic for Model Data Alignment

2025/11/26 by Salgado, Henry, Kendall, Meagan R., Ceberio, Martine
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2511.21931

Abstract

In this work, we propose a simple and computationally efficient framework for evaluating whether machine learning models align with the structure of the data they learn from; that is, whether the model says what the data says. Unlike existing interpretability methods that focus exclusively on explaining model behavior, our approach establishes a baseline derived directly from the data itself. Drawing inspiration from Rubin's Potential Outcomes Framework, we quantify how strongly each feature separates the two outcome groups in a binary classification task, moving beyond traditional descriptive statistics to estimate each feature's effect on the outcome. By comparing these data-derived feature rankings with model-based explanations, we provide practitioners with an interpretable and model-agnostic method for assessing model-data alignment.

Citations

• Reasoning Models Don't Always Say What They Think
• General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Estimating causal effects of treatments in randomized and nonrandomized studies.

Related

• Manipulating and Measuring Model Interpretability
• An unexpected unity among methods for interpreting model predictions
• Impact of Accuracy on Model Interpretations
• Model-Agnostic Interpretation Framework in Machine Learning: A Comparative Study in NBA Sports
• Model-Agnostic Interpretability of Machine Learning