2013/08/26 by Matt Taddy, Taddy, Matt
Computer Science · Mathematics · #Advanced Statistical Methods and Models #Applications (stat.AP) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Statistical Methods and Inference
paper · pdf · doi:10.48550/arxiv.1308.5623
openalex publication_date 2013/08/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The statistics literature of the past 15 years has established many favorable properties for sparse diminishing-bias regularization: techniques which can roughly be understood as providing estimation under penalty functions spanning the range of concavity between L0 and L1 norms. However, lasso L1-regularized estimation remains the standard tool for industrial `Big Data' applications because of its minimal computational cost and the presence of easy-to-apply rules for penalty selection. In response, this article proposes a simple new algorithm framework that requires no more computation than a lasso path: the path of one-step estimators (POSE) does L1 penalized regression estimation on a grid of decreasing penalties, but adapts coefficient-specific weights to decrease as a function of the coefficient estimated in the previous path step. This provides sparse diminishing-bias regularization at no extra cost over the fastest lasso algorithms. Moreover, our `gamma lasso' implementation of POSE is accompanied by a reliable heuristic for the fit degrees of freedom, so that standard information criteria can be applied in penalty selection. We also provide novel results on the distance between weighted-L1 and L0 penalized predictors; this allows us to build intuition about POSE and other diminishing-bias regularization schemes. The methods and results are illustrated in extensive simulations and in application of logistic regression to evaluating the performance of hockey players.