2024/11/19 by Ivy Zhang, Robert Tibshirani, Zhang, Ivy +1
Computer Science · Engineering · #FOS: Computer and information sciences #Flow Measurement and Analysis #Gaussian Processes and Bayesian Inference #Methodology (stat.ME) #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2411.12294
openalex publication_date 2024/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
This paper proposes a sparse regression method that continuously interpolates between Forward Stepwise selection (FS) and the LASSO. When tuned appropriately, our solutions are much sparser than typical LASSO fits but, unlike FS fits, benefit from the stabilizing effect of shrinkage. Our method, Adaptive Forward Stepwise Regression (AFS) addresses this need for sparser models with shrinkage. We show its connection with boosting via a soft-thresholding viewpoint and demonstrate the ease of adapting the method to classification tasks. In both simulations and real data, our method has lower mean squared error and fewer selected features across multiple settings compared to popular sparse modeling procedures.