Fast and Scalable Score-Based Kernel Calibration Tests

2025/10/16 by Glaser, Pierre, Widmann, David, Lindsten, Fredrik +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2510.14711

Abstract

We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a non-parametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a conditional goodness-of-fit criterion for the KCCSD test's U-statistic. We demonstrate the properties of our test on various synthetic settings.

Citations

• Towards Healing the Blindness of Score Matching
• Variational methods for simulation-based inference
• T-Cal: An optimal test for the calibration of predictive models
• Distribution Regression with Sliced Wasserstein Kernels
• Stein's Method Meets Computational Statistics: A Review of Some Recent Developments
• The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning
• Blindness of score-based methods to isolated components and mixing proportions
• Testing Goodness of Fit of Conditional Density Models with Kernels
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Generative Modeling by Estimating Gradients of the Data Distribution
• Distribution Calibration for Regression
• Evaluating model calibration in classification
• Fast Calibrated Additive Quantile Regression
• On Calibration of Modern Neural Networks
• Measuring Sample Quality with Kernels
• Measuring Sample Quality with Diffusions
• A Kernel Test of Goodness of Fit
• Density Estimation in Infinite Dimensional Exponential Families
• Interpretation and Generalization of Score Matching
• Derivative reproducing properties for kernel methods in learning theory
• Vector valued reproducing kernel Hilbert spaces and universality
• Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality
• Maximum Likelihood Learning of Unnormalized Models for Simulation-Based Inference
• Computing and Graphing Highest Density Regions

Cited by

• Kernel-Based Evaluation of Conditional Biological Sequence Models

Related

• Standardisation-function Kernel Stein Discrepancy: A Unifying View on Kernel Stein Discrepancy Tests for Goodness-of-fit
• Semiparametric KSD test: unifying score and distance-based approaches for goodness-of-fit testing
• Using Perturbation to Improve Goodness-of-Fit Tests based on Kernelized Stein Discrepancy
• All Models Are Miscalibrated, But Some Less So: Comparing Calibration with Conditional Mean Operators
• Kernelized Complete Conditional Stein Discrepancy