Minimax Optimal Estimation of KL Divergence for Continuous Distributions

2020/02/26 by Puning Zhao, Zhao, Puning, Lifeng Lai +1 · 2 citations
Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #Face and Expression Recognition #Information Theory (cs.IT) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2002.11599

openalex publication_date 2020/02/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.

Citations

• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation

Cited by

• Dimension Agnostic Testing of Survey Data Credibility through the Lens of Regression
• Deviation Inequalities for Rényi Divergence Estimators via Variational Expression

Related

• Estimation of KL Divergence: Optimal Minimax Rate
• Nearly Minimax Discrete Distribution Estimation in Kullback-Leibler Divergence with High Probability
• A New Lower Bound for Kullback-Leibler Divergence Based on Hammersley-Chapman-Robbins Bound
• On Estimating L₂² Divergence
• A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds