2009/08/13 by Guilherme V. Rocha, Xing Wang, Rocha, Guilherme V. +3 · 1 citation
Economics, Econometrics and Finance · Mathematics · #62F12 #62J07 #62J12 #FOS: Mathematics #Financial Risk and Volatility Modeling #Statistical Distribution Estimation and Applications #Statistical Methods and Inference #Statistics Theory (math.ST) #math.ST #msc:62F12 #msc:62J07 #msc:62J12 #stat.TH
paper · pdf · doi:10.48550/arxiv.0908.1940
55 pages, 4 figures, also available as a technical report from the Statistics Department at Indiana University
arxiv created 2009/08/13 · openalex publication_date 2009/08/13 · arxiv updated 2009/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Since its early use in least squares regression problems, the l1-penalization framework for variable selection has been employed in conjunction with a wide range of loss functions encompassing regression, classification and survival analysis. While a well developed theory exists for the l1-penalized least squares estimates, few results concern the behavior of l1-penalized estimates for general loss functions. In this paper, we derive two results concerning penalized estimates for a wide array of penalty and loss functions. Our first result characterizes the asymptotic distribution of penalized parametric M-estimators under mild conditions on the loss and penalty functions in the classical setting (fixed-p-large-n). Our second result explicits necessary and sufficient generalized irrepresentability (GI) conditions for l1-penalized parametric M-estimates to consistently select the components of a model (sparsistency) as well as their sign (sign consistency). In general, the GI conditions depend on the Hessian of the risk function at the true value of the unknown parameter. Under Gaussian predictors, we obtain a set of conditions under which the GI conditions can be re-expressed solely in terms of the second moment of the predictors. We apply our theory to contrast l1-penalized SVM and logistic regression classifiers and find conditions under which they have the same behavior in terms of their model selection consistency (sparsistency and sign consistency). Finally, we provide simulation evidence for the theory based on these classification examples.