Non-Asymptotic Guarantees for Robust Statistical Learning under Infinite Variance Assumption

2022/01/10 by Xu, Lihu, Yao, Fang, Yao, Qiuran +1 · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2201.03182

Abstract

There has been a surge of interest in developing robust estimators for models with heavy-tailed and bounded variance data in statistics and machine learning, while few works impose unbounded variance. This paper proposes two type of robust estimators, the ridge log-truncated M-estimator and the elastic net log-truncated M-estimator. The first estimator is applied to convex regressions such as quantile regression and generalized linear models, while the other one is applied to high dimensional non-convex learning problems such as regressions via deep neural networks. Simulations and real data analysis demonstrate the robustness of log-truncated estimations over standard estimations.

Cited by

• Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards

Related

• Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results
• New Empirical Process Tools and Their Applications to Robust Deep ReLU Networks and Phase Transitions for Nonparametric Regression
• Robust Nonparametric Regression with Deep Neural Networks
• High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality
• High-Dimensional Learning under ApproximateSparsity with Applications to Nonsmooth Estimation and Regularized Neural Networks