Occam factors and model independent Bayesian learning of continuous distributions

2000/09/30 by Ilya Nemenman, William Bialek · 2 citations
Computer Science · Physics and Astronomy · #Bayesian Methods and Mixture Models #Gaussian Processes and Bayesian Inference #Machine Learning and Algorithms #cond-mat #cs.LG #nlin.AO #physics.data-an

paper · pdf · doi:10.1103/physreve.65.026137

published as Phys. Rev. E, 65 (2), 2002 · publication revisions: extended introduction, new references, other minor corrections; 6 pages, 6 figures, revtex

openalex publication_date 2002/01/24 · arxiv created 2002/02/05 · arxiv updated 2009/11/30 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Learning of a smooth but nonparametric probability density can be regularized using methods of quantum field theory. We implement a field theoretic prior numerically, test its efficacy, and show that the data and the phase space factors arising from the integration over the model space determine the free parameter of the theory ("smoothness scale") self-consistently. This persists even for distributions that are atypical in the prior and is a step towards a model independent theory for learning continuous distributions. Finally, we point out that a wrong parametrization of a model family may sometimes be advantageous for small data sets.

Cited by

• Bayesian Field Theory of the Rate Estimation
• Estimation of probability densities using scale-free field theories

Related

• Bayesian Field Theory: Nonparametric Approaches to Density Estimation, Regression, Classification, and Inverse Quantum Problems
• Scaling Up Machine Learning For Quantum Field Theory with Equivariant Continuous Flows
• Occam's Ghost
• Statistical Inference, Occam's Razor and Statistical Mechanics on The Space of Probability Distributions
• Regularization by Test Function