Finite-sample risk bounds for maximum likelihood estimation with arbitrary penalties

2017/12/29 by W. D. Brinda, Brinda, W. D., Jason M. Klusowski +1
Mathematics · #62B10 94A20 94A15 62E17 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Statistics Theory (math.ST) #math.ST #msc:62B10 #msc:62E17 #msc:94A15 #msc:94A20 #stat.ML #stat.TH

paper · pdf · doi:10.48550/arxiv.1712.10087

To appear in IEEE Transactions on Information Theory, 2018

arxiv created 2017/12/29 · arxiv updated 2018/01/01

Abstract

The MDL two-part coding index of resolvability provides a finite-sample upper bound on the statistical risk of penalized likelihood estimators over countable models. However, the bound does not apply to unpenalized maximum likelihood estimation or procedures with exceedingly small penalties. In this paper, we point out a more general inequality that holds for arbitrary penalties. In addition, this approach makes it possible to derive exact risk bounds of order 1/n for iid parametric models, which improves on the order (log n)/n resolvability bounds. We conclude by discussing implications for adaptive estimation.

Related

• Information Theory of Penalized Likelihoods and its Statistical Implications
• Finite sample expansions and risk bounds in high-dimensional SLS models
• A penalized exponential risk bound in parametric estimation
• Quasi-maximum likelihood estimation and penalized estimation under non-standard conditions
• Adaptive Confidence Sets for the Optimal Approximating Model