2023/12/13 by Mona Schirmer, Dan Zhang, Schirmer, Mona +3
Computer Science · Decision Sciences · Mathematics · #Advanced Statistical Methods and Models #Artificial Intelligence (cs.AI) #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Forecasting Techniques and Applications #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2312.08033
openalex publication_date 2023/12/13 · openalex created_date 2023/12/15 · openalex updated_date 2026/07/28
Knowing if a model will generalize to data 'in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback-Leibler divergence. We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.