2013/12/31 by Justin B. Kinney · 23 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · Physics and Astronomy · #Algorithm #Computer science #Density estimation #Field (mathematics) #Interface (matter) #Mathematical analysis #Mathematics #Occam's razor #Physics #Scale (ratio) #Smoothness #Software #Spectroscopy and Quantum Chemical Studies #Statistical Mechanics and Entropy #Statistical physics #Statistics #Theoretical and Computational Physics #cs.LG #math.ST #occam #physics.data-an #q-bio.QM #stat.ML #stat.TH
paper · pdf · doi:10.1103/physreve.90.011301
published in Physical Review E 90(1), 011301 (American Physical Society) · 4 pages, 4 figures. Major revision in v3. The "Density Estimation using Field Theory" (DEFT) software package is available at https://github.com/jbkinney/13_deft
arxiv created 2014/04/18 · openalex publication_date 2014/07/11 · arxiv updated 2014/07/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
The question of how best to estimate a continuous probability density from finite data is an intriguing open problem at the interface of statistics and physics. Previous work has argued that this problem can be addressed in a natural way using methods from statistical field theory. Here I describe results that allow this field-theoretic approach to be rapidly and deterministically computed in low dimensions, making it practical for use in day-to-day data analysis. Importantly, this approach does not impose a privileged length scale for smoothness of the inferred probability density, but rather learns a natural length scale from the data due to the tradeoff between goodness of fit and an Occam factor. Open source software implementing this method in one and two dimensions is provided.