Benchmarking Simulation-Based Inference

2021/01/12 by Jan-Matthis Lueckmann, Lueckmann, Jan-Matthis, Jan Boelts +7 · 16 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Markov Chains and Monte Carlo Methods

paper · pdf · doi:10.48550/arxiv.2101.04653

openalex publication_date 2021/01/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.

Citations

• Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
• Statistical inference for noisy nonlinear ecological dynamic systems
• Julia: A Fresh Approach to Numerical Computing
• Approximating the Likelihood in Approximate Bayesian Computation
• Validating Bayesian Inference Algorithms with Simulation-Based Calibration
• Automatic Posterior Transformation for Likelihood-Free Inference
• Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On Contrastive Learning for Likelihood-free Inference

Cited by

• Amortized Inference of Multi-Modal Posteriors using Likelihood-Weighted Normalizing Flows
• Simulation and inference methods for non-Markovian stochastic biochemical reaction networks
• Selecting Belief-State Approximations in Simulators with Latent States
• Fast and Robust Simulation-Based Inference With Optimization Monte Carlo
• A Comparison of Kernels for ABC-SMC
• Likelihood-free inference of phylogenetic tree posterior distributions
• Simulation-based inference at the theoretical limit for fast, robust microstructural MRI with minimal diffusion data
• Conditional Flow Matching for Bayesian Posterior Inference
• metabeta -- A fast neural model for Bayesian mixed-effects regression
• One-shot Conditional Sampling: MMD meets Nearest Neighbors
• Improving Generative Methods for Causal Evaluation via Simulation-Based Inference
• CIGaRS I: Combined simulation-based inference from SNae Ia and host photometry
• Trustworthy scientific inference with generative models
• Quantitative and Predictive Folding Models from Limited Single-Molecule Data Using Simulation-Based Inference
• Simulation-based inference for Precision Neutrino Physics through Neural Monte Carlo tuning
• Estimating Marginal Likelihoods in Likelihood-Free Inference via Neural Density Estimation

Related

• sbi reloaded: a toolkit for simulation-based inference workflows
• Simulation-Based Inference: A Practical Guide
• SBI -- A toolkit for simulation-based inference
• An Introduction to Bayesian and Frequentist Simulation-Based Inference with Machine Learning
• Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models