The exact form of the 'Ockham factor' in model selection

2019/06/27 by Rougier, Jonathan, Priebe, Carey
#FOS: Computer and information sciences #FOS: Mathematics #Methodology (stat.ME) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.1906.11592

Abstract

We explore the arguments for maximizing the `evidence' as an algorithm for model selection. We show, using a new definition of model complexity which we term `flexibility', that maximizing the evidence should appeal to both Bayesian and Frequentist statisticians. This is due to flexibility's unique position in the exact decomposition of log-evidence into log-fit minus flexibility. In the Gaussian linear model, flexibility is asymptotically equal to the Bayesian Information Criterion (BIC) penalty, but we caution against using BIC in place of flexibility for model selection.

Related

• Occam Factor for Gaussian Models With Unknown Variance Structure
• Easy computation of the Bayes Factor to fully quantify Occam's razor
• On the overestimation of widely applicable Bayesian information criterion
• Information-based inference for singular models and finite sample sizes: A frequentist information criterion
• Multimodel Inference