Better Subset Regression Using the Nonnegative Garrote

1995/11/01 by Leo Breiman · 631 citations
Engineering · Mathematics · #Advanced Statistical Methods and Models #Applied mathematics #Artificial intelligence #Computer science #Control Systems and Identification #Geology #Instability #Linear regression #Machine learning #Mathematics #Mean squared prediction error #Regression #Regression analysis #Ridge #Selection (genetic algorithm) #Stability (learning theory) #Statistical Methods and Inference #Statistics #Stepwise regression

paper · doi:10.1080/00401706.1995.10484371

published in Technometrics 37(4), 373-384 (Taylor & Francis)

openalex publication_date 1995/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23

Abstract

A new method, called the nonnegative (nn) garrote, is proposed for doing subset regression. It both shrinks and zeroes coefficients. In tests on real and simulated data, it produces lower prediction error than ordinary subset selection. It is also compared to ridge regression. If the regression equations generated by a procedure do not change drastically with small changes in the data, the procedure is called stable. Subset selection is unstable, ridge is very stable, and the nn-garrote is intermediate. Simulation results illustrate the effects of instability on prediction error.

Citations

• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• A Statistical View of Some Chemometrics Regression Tools

Cited by

• Least angle regression
• Wavelet Shrinkage Denoising Using the Non-Negative Garrote
• Sample size for binary logistic prediction models: Beyond events per variable criteria
• Practical variable selection for generalized additive models
• Local partial-likelihood estimation for lifetime data
• A data-driven block thresholding approach to wavelet estimation
• Pathwise coordinate optimization
• A Variational View on Statistical Multiscale Estimation
• A Model-free Variable Screening Method Based on Leverage Score
• Nearest shrunken centroids via alternative genewise shrinkages. [europepmc]
• Genomic Prediction of Breeding Values Using a Subset of SNPs Identified by Three Machine Learning Methods. [europepmc]
• Statistical predictions with glmnet. [europepmc]
• Correntropy-Induced Discriminative Nonnegative Sparse Coding for Robust Palmprint Recognition. [europepmc]
• Penalized Quadratic Inference Function-Based Variable Selection for Generalized Partially Linear Varying Coefficient Models with Longitudinal Data. [europepmc]
• A Radiomics Signature to Quantitatively Analyze COVID-19-Infected Pulmonary Lesions. [europepmc]
• Predicting Grating Orientations With Cross-Frequency Coupling and Least Absolute Shrinkage and Selection Operator in V1 and V4 of Rhesus Monkeys. [europepmc]
• Boosting and lassoing new prostate cancer SNP risk factors and their connection to selenium. [europepmc]
• Regularized Ordinal Regression and the ordinalNet R Package. [europepmc]
• Multivariate Functional Kernel Machine Regression and Sparse Functional Feature Selection. [europepmc]
• Variable selection in elliptical linear mixed model. [europepmc]
• Identifying COVID-19 Severity-Related SARS-CoV-2 Mutation Using a Machine Learning Method. [europepmc]
• Robust model selection using the out-of-bag bootstrap in linear regression. [europepmc]
• Research on Nickel Material Trade Redistribution Strategy Based on the Maximum Entropy Principle. [europepmc]
• Identifying MicroRNA Markers That Predict COVID-19 Severity Using Machine Learning Methods. [europepmc]
• Identification of Methylation Signatures and Rules for Sarcoma Subtypes by Machine Learning Methods. [europepmc]
• Regression shrinkage and selection via least quantile shrinkage and selection operator. [europepmc]
• Deep learning-based EEG emotion recognition: Current trends and future perspectives. [europepmc]
• Identification of Genes Associated with the Impairment of Olfactory and Gustatory Functions in COVID-19 via Machine-Learning Methods. [europepmc]
• Identification of dynamic gene expression profiles during sequential vaccination with ChAdOx1/BNT162b2 using machine learning methods. [europepmc]
• Modeling Categorical Variables by Mutual Information Decomposition. [europepmc]
• Machine Learning Classification of Time since BNT162b2 COVID-19 Vaccination Based on Array-Measured Antibody Activity. [europepmc]
• Regression calibration utilizing biomarkers developed from high-dimensional metabolites. [europepmc]
• Prognostic and chemotherapeutic implications of a novel four-gene pyroptosis model in head and neck squamous cell carcinoma. [europepmc]
• Identification of Risk Factors for Relapse in Childhood Leukemia Using Penalized Semi-parametric Mixture Cure Competing Risks Model. [europepmc]

Related

• Exhuming nonnegative garrote from oblivion using suitable initial estimates- illustration in low and high-dimensional real data
• Automated Analysis of Experiments using Hierarchical Garrote
• Wavelet Shrinkage Denoising Using the Non-Negative Garrote
• Best Subset Selection in Reduced Rank Regression
• Non-null Shrinkage Regression and Subset Selection via the Fractional Ridge Regression