Non-null Shrinkage Regression and Subset Selection via the Fractional Ridge Regression

2025/05/29 by Leonard A. Stefanski, Park, Sihyung, Stefanski, Leonard A.
Decision Sciences · #Grey System Theory Applications

paper · pdf · doi:10.48550/arxiv.2505.23925

Abstract

ℓ_p-norm penalization, notably the Lasso, has become a standard technique, extending shrinkage regression to subset selection. Despite aiming for oracle properties and consistent estimation, existing Lasso-derived methods still rely on shrinkage toward a null model, necessitating careful tuning parameter selection and yielding monotone variable selection. This research introduces Fractional Ridge Regression, a novel generalization of the Lasso penalty that penalizes only a fraction of the coefficients. Critically, Fridge shrinks the model toward a non-null model of a prespecified target size, even under extreme regularization. By selectively penalizing coefficients associated with less important variables, Fridge aims to reduce bias, improve performance relative to the Lasso, and offer more intuitive model interpretation while retaining certain advantages of best subset selection.

Citations

• Elastic Net Regularization Paths for All Generalized Linear Models
• Elastic Net Regularization Paths for All Generalized Linear Models
• Strong rules for discarding predictors in lasso-type problems
• Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems
• Safe Feature Elimination in Sparse Supervised Learning
• Nearly unbiased variable selection under minimax concave penalty
• Ridge Regression: Biased Estimation for Nonorthogonal Problems

Related

• Lasso and equivalent quadratic penalized models
• Post Selection Shrinkage Estimation for High Dimensional Data Analysis
• Subset Selection with Shrinkage: Sparse Linear Modeling when the SNR is low
• Lasso-Ridge Refitting: A Two-Stage Estimator for High-Dimensional Linear Regression
• Adaptive Shrinkage with a Nonparametric Bayesian Lasso