The composite absolute penalties family for grouped and hierarchical variable selection

2009/08/17 by Peng Zhao, Guilherme Rocha, Guilherme V. Rocha +1 · 652 citations
Decision Sciences · Engineering · Mathematics · #Algorithm #Artificial intelligence #Computer science #Feature selection #Lasso (programming language) #Mathematical optimization #Mathematics #Mean squared error #Model selection #Probabilistic and Robust Engineering Design #Regression #Regularization (linguistics) #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #Statistics #math.ST #msc:62J07 #stat.TH

paper · pdf · doi:10.1214/07-aos584

published in The Annals of Statistics 37(6A) (Institute of Mathematical Statistics) · Published in at http://dx.doi.org/10.1214/07-AOS584 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

openalex publication_date 2009/08/17 · arxiv created 2009/09/02 · arxiv updated 2009/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Extracting useful information from high-dimensional data is an important focus of today’s statistical research and practice. Penalized loss function minimization has been shown to be effective for this task both theoretically and empirically. With the virtues of both regularization and sparsity, the L1-penalized squared error minimization method Lasso has been popular in regression models and beyond. In this paper, we combine different norms including L1 to form an intelligent penalty in order to add side information to the fitting of a regression or classification model to obtain reasonable estimates. Specifically, we introduce the Composite Absolute Penalties (CAP) family, which allows given grouping and hierarchical relationships between the predictors to be expressed. CAP penalties are built by defining groups and combining the properties of norm penalties at the across-group and within-group levels. Grouped selection occurs for nonoverlapping groups. Hierarchical variable selection is reached by defining groups with particular overlapping patterns. We propose using the BLASSO and cross-validation to compute CAP estimates in general. For a subfamily of CAP estimates involving only the L1 and L∞ norms, we introduce the iCAP algorithm to trace the entire regularization path for the grouped selection problem. Within this subfamily, unbiased estimates of the degrees of freedom (df) are derived so that the regularization parameter is selected without cross-validation. CAP is shown to improve on the predictive performance of the LASSO in a series of simulated experiments, including cases with p≫n and possibly mis-specified groupings. When the complexity of a model is properly calculated, iCAP is seen to be parsimonious in the experiments.

Citations

• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
• Atomic Decomposition by Basis Pursuit
• A Statistical View of Some Chemometrics Regression Tools
• Ideal spatial adaptation by wavelet shrinkage
• Ridge Regression: Biased Estimation for Nonorthogonal Problems

Cited by

• Variational approximate penalized credible regions for Bayesian grouped regression
• High-Dimensional Covariance Decomposition into Sparse Markov and Independence Models
• The Knowledge Gradient Policy Using A Sparse Additive Belief Model
• Sparse Partially Linear Additive Models
• Regularization methods for solving hierarchical variational inequalities with complexity guarantees
• Moving Object Detection by Detecting Contiguous Outliers in the Low-Rank Representation
• Neural Architecture Search as Sparse Supernet
• Regularization for Multiple Kernel Learning via Sum-Product Networks
• Proximal methods for the latent group lasso penalty
• Variable selection and regularization via arbitrary rectangle-range generalized elastic net
• Sparse hierarchical interaction learning with epigraphical projection
• Variable Selection in Regression-based Estimation of Dynamic Treatment Regimes
• Safe Screening Rules for Group SLOPE
• Modeling high dimensional multichannel brain signals
• Penalized Cox’s proportional hazards model for high-dimensional survival data with grouped predictors
• Learning with Optimal Interpolation Norms
• Sparse Deep Additive Model with Interactions: Enhancing Interpretability and Predictability
• Model Selection for High Dimensional Quadratic Regression via Regularization
• Individual Data Protected Integrative Regression Analysis of High-dimensional Heterogeneous Data
• Parametric Maxflows for Structured Sparse Learning with Convex Relaxations of Submodular Functions
• Tikhonov regularized exterior penalty methods for hierarchical variational inequalities
• Fitting ARMA Time Series Models without Identification: A Proximal Approach
• Smoothing Proximal Gradient Method for General Structured Sparse Learning
• High-dimensional Mixed Graphical Models
• Reluctant Interaction Modeling
• Learning with Structured Sparsity
• Learning Efficient Structured Sparse Models
• Screening Rules for Overlapping Group Lasso
• Structured Sparse Aggregation
• Bayesian group latent factor analysis with structured sparsity
• Distributed Multitask Learning
• Learning Undirected Graphical Models with Structure Penalty
• LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models
• From Graphical Lasso to Atomic Norms: High-Dimensional Pattern Recovery
• Multimodal Sparse Bayesian Dictionary Learning
• The BerHu penalty and the grouped effect
• Parallel Direction Method of Multipliers
• Proximal-Proximal-Gradient Method
• Graph-based regularization for regression problems with alignment and highly-correlated designs
• Nonparametric sparsity and regularization
• Exploiting Correlation in Sparse Signal Recovery Problems: Multiple Measurement Vectors, Block Sparsity, and Time-Varying Sparsity
• Main and Interaction Effects Selection for Quadratic Discriminant Analysis via Penalized Linear Regression
• Convex Banding of the Covariance Matrix
• Collaborative Multi-sensor Classification via Sparsity-based Representation
• Joint Estimation of Precision Matrices in Heterogeneous Populations
• Model Selection for High-Dimensional Quadratic Regression via Regularization
• Fast projections onto mixed-norm balls with applications
• Multi-scale Mining of fMRI data with Hierarchical Structured Sparsity
• Pathways-driven Sparse Regression Identifies Pathways and Genes Associated with High-density Lipoprotein Cholesterol in Two Asian Cohorts
• A Selective Review of Group Selection in High-Dimensional Models
• Efficient algorithm to select tuning parameters in sparse regression modeling with regularization
• Prediction of hierarchical time series using structured regularization and its application to artificial neural networks
• Estimation of Sparse MIMO Channels with Common Support
• Generalized Conditional Gradient for Sparse Estimation

Related

• Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
• Collinear Groupwise Selection via Scaled Group Lasso
• Nearly unbiased variable selection under minimax concave penalty
• Automatic selection by penalized asymmetric Lq-norm in an high-dimensional model with grouped variables
• Group Regularized Estimation under Structural Hierarchy