2008/03/01 by Tong Tong Wu, Kenneth Lange · 1 citation
Computer Science · Mathematics · #Coordinate descent #Elastic net regularization #Euclidean distance #Gaussian Processes and Bayesian Inference #Greedy algorithm #Lasso (programming language) #Linear regression #Norm (philosophy) #Regression #Regression analysis #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #stat.AP
paper · pdf · doi:10.1214/07-aoas147
published as Annals of Applied Statistics 2008, Vol. 2, No. 1, 224-244 · Published in at http://dx.doi.org/10.1214/07-AOAS147 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)
openalex publication_date 2008/03/01 · arxiv created 2008/03/27 · arxiv updated 2009/12/01 · openalex created_date 2016/06/24 · openalex updated_date 2026/08/06
Imposition of a lasso penalty shrinks parameter estimates toward zero and performs continuous model selection. Lasso penalized regression is capable of handling linear regression problems where the number of predictors far exceeds the number of cases. This paper tests two exceptionally fast algorithms for estimating regression coefficients with a lasso penalty. The previously known ℓ2 algorithm is based on cyclic coordinate descent. Our new ℓ1 algorithm is based on greedy coordinate descent and Edgeworth’s algorithm for ordinary ℓ1 regression. Each algorithm relies on a tuning constant that can be chosen by cross-validation. In some regression problems it is natural to group parameters and penalize parameters group by group rather than separately. If the group penalty is proportional to the Euclidean norm of the parameters of the group, then it is possible to majorize the norm and reduce parameter estimation to ℓ2 regression with a lasso penalty. Thus, the existing algorithm can be extended to novel settings. Each of the algorithms discussed is tested via either simulated or real data or both. The Appendix proves that a greedy form of the ℓ2 algorithm converges to the minimum value of the objective function.