Predictive Multiplicity in Classification

2019/09/14 by Charles T. Marx, Marx, Charles T., Flavio du Pin Calmon +4 · 28 citations
Computer Science · Mathematics · #Bayesian Modeling and Causal Inference #Computability, Logic, AI Algorithms #Computers and Society (cs.CY) #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.CY #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1909.06677

openalex publication_date 2019/09/14 · arxiv created 2020/09/16 · arxiv updated 2020/09/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with conflicting predictions. We introduce formal measures to evaluate the severity of predictive multiplicity and develop integer programming tools to compute them exactly for linear classification problems. We apply our tools to measure predictive multiplicity in recidivism prediction problems. Our results show that real-world datasets may admit competing models that assign wildly conflicting predictions, and motivate the need to measure and report predictive multiplicity in model development.

Citations

• Explaining Data-Driven Document Classifications1
• Prediction Policy Problems
• Interventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment
• Does mitigating ML's impact disparity require treatment disparity?
• The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
• Aequitas: A Bias and Fairness Audit Toolkit
• Fairwashing: the risk of rationalization

Cited by

• EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs
• The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning
• Exploring the Rashomon Set for Concept-Based Models
• Model Agreement via Anchoring
• SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration
• Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicity
• Revisiting the Relation Between Robustness and Universality
• Doctor Rashomon and the UNIVERSE of Madness: Variable Importance with Unobserved Confounding and the Rashomon Effect
• Optimal Robust Recourse with L^p-Bounded Model Change
• Explainable AI Isn't Enough! Rethinking Algorithmic Contestability
• "A 6 or a 9?": Ensemble Learning Through the Multiplicity of Performant Models and Explanations
• Task complexity shapes internal representations and robustness in neural networks
• Observational Multiplicity
• VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance
• On Arbitrary Predictions from Equally Valid Models
• Transformers Don't In-Context Learn Least Squares Regression
• What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
• Statistical Inference for Responsiveness Verification
• GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems
• Argumentative Ensembling for Robust Recourse under Model Multiplicity
• Representation Consistency for Accurate and Coherent LLM Answer Aggregation
• Leveraging Predictive Equivalence in Decision Trees
• Characterizing Fairness Over the Set of Good Models Under Selective Labels
• Selecting for Less Discriminatory Algorithms: A Relational Search Framework for Navigating Fairness-Accuracy Trade-offs in Practice
• Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity
• Rashomon Sets and Model Multiplicity in Federated Learning
• Predictive Multiplicity in Survival Models: A Method for Quantifying Model Uncertainty in Predictive Maintenance Applications
• Regretful Decisions under Label Noise

Related

• Predictive Multiplicity in Probabilistic Classification
• Rashomon Capacity: A Metric for Predictive Multiplicity in Classification
• Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicity
• Observational Multiplicity
• The Role of Hyperparameters in Predictive Multiplicity