Improving the Accuracy of Amortized Model Comparison with Self-Consistency

2025/08/28 by Šimon Kucharský, Kucharský, Šimon, Aayush Mishra +7 · 1 citation
Decision Sciences · #Computation (stat.CO) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2508.20614

openalex publication_date 2025/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Amortized Bayesian model comparison (BMC) enables fast probabilistic ranking of models via simulation-based training of neural surrogates. However, the accuracy of neural surrogates deteriorates when simulation models are misspecified; the very case where model comparison is most needed. We evaluate four different amortized BMC methods. We supplement traditional simulation-based training of these methods with a self-consistency (SC) loss on unlabeled real data to improve BMC estimates under distribution shifts. Using one artificial and two real-world case studies, we compare amortized BMC estimators with and without SC against analytic or bridge sampling benchmarks. In the closed-world case (data is generated by one of the candidate models), BMC estimators using classifiers work acceptably well even without SC training. However, these methods also benefit the least from SC training. In the open-world scenario (all models misspecified), SC training strongly improves BMC estimators when having access to analytic likelihoods, or when surrogate likelihoods are locally accurate near the true parameter posterior, even for severely misspecified models. We conclude with practical recommendations for amortized BMC and suggestions for future research.

Citations

• Robust Amortized Bayesian Inference with Self-Consistency Losses on Unlabeled Data
• The Statistical Accuracy of Neural Posterior and Likelihood Estimation
• Amortized Bayesian Multilevel Models
• On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows
• Validation and Comparison of Non-Stationary Cognitive Models: A Diffusion Model Application
• SimSIMS: Simulation-based Supernova Ia Model Selection with thousands of latent variables
• BayesFlow: Amortized Bayesian Workflows With Neural Networks
• Simultaneous identification of models and parameters of scientific simulators
• Evidence Networks: simple losses for fast, amortized, neural Bayesian model comparison
• JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models
• A Deep Learning Method for Comparing Bayesian Hierarchical Models
• Bayesian Model Selection, the Marginal Likelihood, and Generalization
• Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks
• Generalized Out-of-Distribution Detection: A Survey
• Generalized Out-of-Distribution Detection: A Survey
• Amortized Bayesian model comparison with evidential deep learning
• BayesFlow: Learning complex stochastic models with invertible neural networks
• Neural Spline Flows
• bridgesampling: An R Package for Estimating Normalizing Constants
• Deep Sets
• Density estimation using Real NVP
• Variational Inference with Normalizing Flows
• Strictly Proper Scoring Rules, Prediction, and Estimation

Cited by

• Scalable Simulation-Based Model Inference with Test-Time Complexity Control

Related

• Improving the Accuracy of Amortized Model Comparison with Self-Consistency
• Robust Amortized Bayesian Inference with Self-Consistency Losses on Unlabeled Data
• Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks: An Extended Investigation
• Robust Bayesian methods using amortized simulation-based inference
• Amortized Bayesian Inference for Models of Cognition