2015/12/02 by Sara van de Geer, van de Geer, Sara, Martin Wainwright +1
Mathematics · #62G05 #FOS: Mathematics #Statistics Theory (math.ST) #math.ST #msc:62G05 #stat.TH
paper · pdf · doi:10.48550/arxiv.1512.00677
27 pages
arxiv created 2016/01/11 · arxiv updated 2016/01/12
Rates of convergence for empirical risk minimizers have been well studied in the literature. In this paper, we aim to provide a complementary set of results, in particular by showing that after normalization, the risk of the empirical minimizer concentrates on a single point. Such results have been established by~\citechatterjee2014new for constrained estimators in the normal sequence model. We first generalize and sharpen this result to regularized least squares with convex penalties, making use of a "direct" argument based on Borell's theorem. We then study generalizations to other loss functions, including the negative log-likelihood for exponential families combined with a strictly convex regularization penalty. The results in this general setting are based on more "indirect" arguments as well as on concentration inequalities for maxima of empirical processes.