2017/11/25 by Daniel Conn, Gang Li, Conn, Daniel +1
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #FOS: Mathematics #Face and Expression Recognition #Primary 62G08 #Statistical Methods and Inference #Statistics Theory (math.ST) #math.ST #msc:62G08 #stat.TH
paper · pdf · doi:10.48550/arxiv.1711.09200
arxiv created 2017/11/25 · openalex publication_date 2017/11/25 · arxiv updated 2017/11/28 · openalex created_date 2017/12/04 · openalex updated_date 2026/07/28
The celebrated Nadaraya-Watson kernel estimator is among the most studied method for nonparametric regression. A classical result is that its rate of convergence depends on the number of covariates and deteriorates quickly as the dimension grows, which underscores the "curse of dimensionality" and has limited its use in high dimensional settings. In this article, we show that when the true regression function is single or multi-index, the effects of the curse of dimensionality may be mitigated for the Nadaraya-Watson kernel estimator. Specifically, we prove that with K-fold cross-validation, the Nadaraya-Watson kernel estimator indexed by a positive semidefinite bandwidth matrix has an oracle property that its rate of convergence depends on the number of indices of the regression function rather than the number of covariates. Intuitively, this oracle property is a consequence of allowing the bandwidths to diverge to infinity as opposed to restricting them all to converge to zero at certain rates as done in previous theoretical studies. Our result provides a theoretical perspective for the use of kernel estimation in high dimensional nonparametric regression and other applications such as metric learning when a low rank structure is anticipated. Numerical illustrations are given through simulations and real data examples.