Spectrally Robust Covariance Shrinkage for Hotelling's T² in High Dimensions

2025/02/04 by Benjamin D. Robinson, Van Latimer, Robinson, Benjamin D. +1
Mathematics · #Advanced Mathematical Physics Problems

paper · pdf · doi:10.48550/arxiv.2502.02006

Abstract

We investigate covariance shrinkage for Hotelling's T² in the regime where the data dimension p and the sample size n grow in a fixed ratio -- without assuming that the population covariance matrix is spiked or well-conditioned. When p/n→ϕ∈ (0,1), we propose a practical finite-sample shrinker that, for any maximum-entropy signal prior and any fixed significance level, (a) asymptotically maximizes power under Gaussian data, and (b) asymptotically saturates the Hanson--Wright lower bound on power in the more general sub-Gaussian case. Our approach is to formulate and solve a variational problem characterizing the optimal limiting shrinker, and to show that our finite-sample method consistently approximates this limit by extending recent local random matrix laws. Empirical studies on simulated and real-world data, including the Crawdad UMich/RSS data set, demonstrate up to a 50% gain in power over leading linear and nonlinear competitors at a significance level of 10^-4.

Citations

• A well-conditioned estimator for large-dimensional covariance matrices
• On the distribution of the largest eigenvalue in principal components analysis
• On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices
• A Distribution-Free M-Estimator of Multivariate Scatter

Related

• An Improvement on the Hotelling T² Test Using the Ledoit-Wolf Nonlinear Shrinkage Estimator
• Eigenvector distributions and optimal shrinkage estimators for large covariance and precision matrices
• Optimal Eigenvalue Shrinkage in the Semicircle Limit
• Regularization in High-Dimensional Regression and Classification via\n Random Matrix Theory
• Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model