2012/07/31 by Guido Montúfar, Guido F. Montufar, Johannes Rauh
Computer Science · Decision Sciences · Mathematics · Physics and Astronomy · #Applied mathematics #Computer science #Entropy (arrow of time) #Gaussian Processes and Bayesian Inference #Geometry #Mathematics #Model Reduction and Neural Networks #Physics #Probabilistic and Robust Engineering Design #Scaling #Statistical physics #Thermodynamics #msc:62B15 #msc:94A17 #stat.ML
paper · pdf · doi:10.14736/kyb-2014-2-0234
published as Kybernetika 50 (2014) 2, p. 234-245 · 13 pages, 3 figures, WUPES'12
arxiv created 2013/02/25 · openalex publication_date 2014/05/22 · arxiv updated 2014/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
We compute the expected value of the Kullback-Leibler divergence to various fundamental statistical models with respect to canonical priors on the probability simplex. We obtain closed formulas for the expected model approximation errors, depending on the dimension of the models and the cardinalities of their sample spaces. For the uniform prior, the expected divergence from any model containing the uniform distribution is bounded by a constant 1 − γ, and for the models that we consider, this bound is approached if the state space is very large and the models ’ dimension does not grow too fast. For Dirichlet priors the expected divergence is bounded in a similar way, if the concentration parameters take reasonable values. These results serve as reference values for more complicated statistical models. 1